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PREFACE 


Bienvenue vers la Nouvelle-Orléans! Welcome to New Orleans! We’ve got quite 
a week planned for you here in the Big Easy. Our program includes 34 refereed 
papers, some of which are the best we’ve ever seen at a LISA conference. We’ve 
got the Invited Talks track again this year. And we welcome two new tracks to 
the LISA family this year: one on Security Administration and one on Network 
Administration. 


We’d like to extend special thanks to Ellie Young who has been responsible for 
coordinating and managing this entire event. Thanks again to Dan Klein for the 
excellent tutorial suite that make LISA the premier system administration 
learning event. 


Rob Kolstad brought his considerable typesetting skills on-board to produce the 
proceedings this year, and it was quite a job. This is the largest and most 
comprehensive set of proceedings ever produced for a LISA conference: 378 
pages of papers. 


Tom Limoncelli and Pat Wilson organized this year’s Invited Talks track where 
an "embarrassment of riches" has produced the best Invited Talks track in years. 


Bill LeFebvre and David Williamson assembled the new Network Administration 
track with some really outstanding content for network administrators. 


Simon Cooper and Cat Okita coordinated the new Security Administration track. 
As security become ever-more important to the system administrator community, 
this track will provide a focused look at the state of security administration. 


Lee Damon did an outstanding job coordinating the Guru-Is-In track. He and his 
gurus are available to help solve those quirky and difficult problems. 


Finally, this conference would not be possible without the considerable talents 
and effort from the USENIX Office and the USENIX Conference Office. Thanks 
to them! 


If you have any questions or problems during the conference, please don’t 
hesitate to grab one of the conference organizers and let us know. 


Have a great week in New Orleans! 
Laissez les jeux commencer! 


Your hosts, 
Remy Evard 
Phil Scarr 
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Theoretical System Administration 


Mark Burgess — Oslo University College 


ABSTRACT 


In order to develop system administration strategies which can best achieve organizations’ 
goals, impartial methods of analysis need to be applied, based on the best information available 
about needs and user practices. This paper draws together several threads of earlier research to 
propose an analytical method for evaluating system administration policies, using statistical 


dynamics and the theory of games. 


Introduction 


System administration includes the planning, 
configuration and maintenance of computer systems. 
The discipline of system administration is traditionally 
founded on the anecdotal experiences of system man- 
agers [1, 2], but this can only be carried so far; formal 
(mathematical) analyses of system administration have 
only recently begun to enable more scientific studies 
to be carried out [3, 4]. A lack of formal methods 
makes it difficult to express objective truths about the 
field, avoiding marketing assertions and the vested 
interests of companies and individuals. The aim of this 
paper is to summarize a mathematical formulation of 
system administration, which can account for a basis 
of empirical evidence, and which provides an objec- 
tive approach to study. This is central to the present 
discussion on developing system administration as a 
formal discipline. 


In previous work by the author and collaborators, 
it has been shown how aspects of the average empiri- 
cal behaviour of systems of computers and users can 
be modelled using fairly straightforward statistical 
ideas [5, 6, 7, 8]. This has allowed a coarse statistical 
model of computer systems to be built, which can be 
used as a backdrop for studies of system administra- 
tion. Previous work by other authors has also 
attempted to look at computer ecosystems in terms of 
differential equations [9]. In the future additional 
mathematical models will, no doubt, be devised in 
order to study other issues. 


One of the obstacles to formulating a complete 
theory of system administration is the complexity of 
interaction between humans and computers. There are 
many variables in a computer system, which are con- 
trolled at distributed locations. Computer systems are 
complex in the sense of having many embedded causal 
relationships and controlling parameters. Computer 
behaviour is strongly affected by human social 
behaviour, and this is often unpredictable. However, 
the central question in any scientific investigation is 
one of balance: can one formulate a quantitative the- 
ory of system administration, which is general enough 
to be widely applicable, but which is specific enough 
to admit analysis? 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


The outline of this paper is as follows. To begin 
the discussion some simplified assumptions about the 
aims of system administration are stated. Next, two 
types of quantitative model, describing a computer 
system interacting with users (possibly via a network), 
are described and the primitive operations which can 
be carried out within the scope of the models are iden- 
tified. The two types of model are referred to as type I 
(passive) and type II (strategic). A method of quantify- 
ing the benefits and flaws of different strategies 
emerges from this discussion. Strategies for system 
administration and for user behaviour may then be for- 
mulated and arranged in a matrix allowing the task of 
administrating a computer system can be described in 
precise game theoretical terms. The primary goal of 
this work is to provide the recipe for performing this 
kind of analysis. 


Basic Assumptions 


Capturing such a complex pursuit as system 
administration in a few simple rules is presumptuous, 
but approximately possible if one focuses on core 
activities. In order to make progress one must agree on 
some specific aims for users and administrators. The 
purpose of defining the aims of the interested parties 
in a computer system, is to come up with a good 
enough abstraction for system management that spe- 
cific issues may be addressed in quantifiable terms. In 
this paper, the word ‘system’ will be taken to mean 
any organized collection of computers interacting with 
a group of users. The assumptions used are these: 

e The aim of the system administrator is to keep 
the system alive and running well so that users 
can perform a maximum amount of useful 
work. 

e The aim of benign users 1s to produce useful 
work using the system. This consumes 
resources. 

e The aim of malicious users is to maximize their 
control over system resources. 


A possible quantitative definition of ‘useful 
work’ is the amount of user-data modified on the com- 
puter system. plus the information transmitted to or 
from remote locations. This can be refined for specific 
purposes. Time spent fighting for control of a 
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damaged machine, or other users, for example, is not 
useful work for normal users. 


This short list of aims does not encompass every 
eventuality, but it establishes a starting point. In addi- 
tion to these points, it is necessary to provide a scheme 
of values about what is subjectively good or bad about 
the system. When are things going well and when are 
things going badly? This is done by specifying a sys- 
tem policy [10, 11]. 

A system policy is a specification of a sys- 

tems configuration and its acceptable pat- 

terns of usage. A complete policy therefore 

affects the basic installation of the system 

and also the way it changes in time due to 

interaction with users. 
A system policy is the pillar of truth and measuring 
stick against which one determines whether system 
activity 1s acceptable or unacceptable. A sufficiently 
complete system policy can also include a complete 
configuration blueprint and thus determine whether 
the state of configuration is acceptable. The central 
theorem, which was found in [3] is: 

A sufficiently complete specification of a sys- 

tem policy leads to the notion of an ideal aver- 

age state for the system. Over time, the ideal 

average state of the system degrades. The aim 

of system administration is to keep the system 

as close to its ideal state as possible. 


The meaning of the theorem is that it identifies 
system administration as a regulatory procedure. This 
idea of regulatory action was originally introduced, 
using the term convergence, in connection with the 
system administration tool cfengine [5, 12, 13]. 
Cfengine is a program used to automate the regulation 
of host state, by making it converge towards its ‘ideal 
state’ with every execution of cfengine. For cfengine, 
the ideal state 1s achieved when every detail of a com- 
puter configuration appears to be correctly imple- 
mented and no changes to the system made by users 
contravene system policy. Thus ‘state’ refers to adher- 
ence to a policy. The cfengine model tums out to be a 
useful starting point for discussing system administra- 
tion, since it offers a detailed and concrete idealization 
of system administration tasks in terms of sequences 
of primitive actions. Currently, cfengine does not 
have a complete picture of system state, at all levels, 
though part of the aim of this kind of work is to 
improve on that situation. However, by basing a study 
on this idea one also obtains, as a side effect, a theo- 
retical evaluation of the model which can be used to 
improve cfengine’s design in the future. 


Policy, State, and Convergence 


Without proving the central theorem in this 
paper, it is helpful to provide a brief explanation of 
how the ideal state 1s constructed, and why it is only 
possible to insist on an average description of ideal- 
ness. 
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State is a snapshot of the condition of a system, 
which results from its current configuration and the 
history of all tasks which have consumed and released 
its resources over time. To picture state, it 1s helpful to 
think of a human analogy. In [6], the analogy with 
human health was drawn. Using another analogy, that 
of evolutionary fitness or adaptation for a purpose, 
host state can be envisioned on an arbitrary scale, 
which makes the ideal state that condition in which the 
system is best able to perform its tasks. As with 
humans, general fitness of a computer system is a 
combination of two parts: a part which is determined 
by inherited properties and a part which is the result of 
its interaction with the environment. 


For humans, the state of fitness would be the 
sum of genetically determined attributes (roughly 
speaking, a policy for the operation of the organism) 
and current physical fitness (the attunement or degra- 
dation resulting from interaction with environment). 
For a computer system, there is a similar duality: state 
refers to a part which is the sum of all configuration 
and policy decisions (basic design quality) and a part 
which results from an interaction with users and net- 
work impulses (input/output). 


Thus state separates into policy and environment. 
The state of a computer system S(t) changes continu- 
ously with time, due mainly to the interaction with the 
environment, but also internally, as a general conse- 
quence of the second law of thermodynamics (a statis- 
tical inference which notes that the number of ways in 
which a system can be disordered is far greater than 
the number of ways it can be ordered (adhering to pol- 
icy), thus any random change is statistically more 
likely to lead to disorder than to order). This is the 
principle of increase of entropy [14, 15]. 


The environment of a computer system can be 
thought of as an external batch of transactions (see 
Figure 1), 1.e., input and output which appears and dis- 
appears as users interact with the system. Each trans- 
action makes use of resources and has the possibility 
of affecting the state S(t) of the system by an amount 
0S(t). The number of transactions is generally large for 
periods of time over which one expects the host state 
to change significantly: transactions last usually mil- 
liseconds, whereas host behaviour is self-similar often 
over days and weeks [7]. 


From empirical studies [7, 8], one has a picture 
of a computer system as having an stable average con- 
dition over periods of time, but fluctuating consider- 
ably in response to specific transactions. In other 
words, over days and weeks, computer resources 
change, but over many weeks the pattern of usage has 
a mean value which shows a stable pattern. At any 
given time, the actual values of resource variables are 
different from the mean, but these differences average 
out to an average condition, or state. If this average 
state of the host is to be maintained near its ideal state 
then one hopes that the fluctuations from the mean 
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OS(t) are small, i.e., the disturbance to the system 
resulting from user interactions results in only a small 
change to the actual state. 


System administration 


Environment 


' i 
b i 
1 i 
| i 
1 i 
! i 
' i 
| i 
! i 
i i 
! i 
1 i 





Resources Users 


Figure 1: System administration is a regulative func- 
tion over time. 


There are thus four ideas of state which need to 


be considered: the ideal average state 5 (2), whose 
existence is implied by policy, the actual average state 
of the system S(t) which is the mean value of 
behaviour over weeks, the actual state of the system 
S(t) and fluctuations from the average state dS(t). All 
of these are functions of time. The latter three are 
related by a time-dependent relation: 

S(t) = S(t) + dS(t). 

In order to speak of an average state, one has to 
say what average means. This has been defined pre- 
cisely, using the theory of dynamical systems in [4] 
and tums out to be a regular arithmetical mean, calcu- 
lated from a sliding window data sample, which 
advances over time. Because computer behaviour 
shows approximate periodic repetition, the average is 
defined in terms of co-cycles, over days and weeks: 
the two sociological influences which have profound 
implications for computer behaviour. Our empirical 
studies, at Oslo, have shown that fluctuations in the 
state must be averaged over a window of at least two 
or three weeks [7, 8] in order to see reliable stable 
behaviour under normal usage. This is the time scale 
over which users repeat their behaviour several times, 
within the framework of daily/weekly cycles. 


Note that the ideal average state is only approxi- 
mately constant (it changes slowly, at the same rate as 
the average changes), whereas the other states change 
with more rapidly time (on the time scale of individual 
interactions with the system). The average ideal state 
changes much more slowly than the actual state, pre- 
cisely because it is averaged over coarser grains of 
time. 
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A notion of idealness can thus only be character- 
ized for an average state because the system is con- 
stantly changing as users interact with it. Even the 
mean value is changing slowly with time. In physics 
of statistical systems, this is referred to as non-equilib- 
rium behaviour. However, the fact that this decompo- 
sition is possible is important. It separates the effects 
of independent scales from one another. What happens 
in the short term is different to what happens on aver- 
age, since one deviation might correct another, leaving 
no net problem. For example, if a user consumes a 
large amount of resources for a brief time (a tempo- 
rary file, for instance) while performing useful work, 
this will only affect the actual state of the system for 
the duration of that task, provided the file is removed 
afterwards. A policy of file temporary file garbage 
collection can always remove such a file even if a user 
doesn’t. The average state will therefore be relatively 
unaffected by short term changes. The meaning of the 
environmental ideal average state is therefore to define 
an interval of stability for interaction with environ- 
ment. The system will always deviate from the so- 
called ideal state, but that need not be a problem as 
long as it does not deviate far from it for long periods 
of time. 


There are two types of disturbance 6S: those 
which (on average) preserve the state of the system, 
1.€., those which release as many resources as they 
consume, and those which consume resources without 
releasing them. The latter kind of disturbance is the 
most dangerous to the integrity of the system, since it 
can lead to runaway behaviour which sees the end of 
the system. This 1s the case in which it is necessary to 
introduce countermeasures to protect the state of the 
system. This is the purpose of computer immunology 
[6]. When large fluctuations are at hand, the system is 
in an intrinsically unstable state. 


How can changes of state be characterized pre- 
cisely? An obvious choice is to use the mathematical 
idea of a lattice, or discrete vector space. Although 
computer behaviour often has the appearance of a con- 
tinuously varying load, the actual changes are all dis- 
crete in nature. Any interactive change in the system 
may be broken down into a sequence of discrete prim- 
itive operations. See Table 1 for the primitives used by 
cfengine [5, 12]. 

Any change to the configurable system, can be 
expressed in terms of these primitives. In addition to 
these, there are kernel variables which contain data 
that can be used to determine environmental state. 
Each independent primitive can be thought of as an 
axis in an n-dimensional lattice. Each change in the 
state of the system, of a given type, is a movement 
through the lattice in that direction. Moreover, since 
the averaging procedure for environment effectively 
divides up time into co-cyclic discrete units, (days and 
weeks) and scaled coarse-grained intervals, it is possi- 
ble to draw the the state of the system on a lattice (see 
Figure 2). Mathematicians note: the lattice is only 
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conformally distorted by changes in the averaging pro- 
cedure; the structure 1s preserved. 


Primitive type 7 Comments/Examples | 


Create file object Touch 

Tidy garbage 
Disable 
Configuration 


Delete file object 
Rename file object 
Edit file 

Edit access control 
Request resource 
Copy file Read/write 
Process control Start/stop 
Process priority Nice 
Configure device | ifconfig/ioctl 


Permissions 
Read/Mount 





Table 1: cfengine primitives. | 


In principle, there is a single point in the lattice 
which represents (at any given time) the most ideal 
state possible. In practice, one is only interested in 
keeping the system in a region, not too far from this 
practically unobtainable ideal point. 


Changes to system policy must also be discrete 
strings of these primitives, since they have to be 


-> ‘ 


0 
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implemented using the primitives, and thus a change 
in policy is simply a translation of the ideal state 
through the lattice. 


Suppose one places the ideal state arbitrarily at 
the origin of this lattice. The further the system devi- 
ates from this origin, the more precarious the state of 
the system. Eventually when the state strays a suffi- 
cient distance from the origin, the system will exhaust 
its resources and fail completely. In the intervening 
distance, the system is working in accordance with 
policy when it is close to the ideal state. Using the 
Euclidean distance as a Hamming distance, for change 
in the system, it is possible to see that the number of 
corrective actions for required to return the system to 
its ideal state grows only linearly, however the number 
of possible corrective procedures increases exponen- 
tially. 


The number of equivalent paths H(d) back to the 
ideal state 1s: 


(34)! 


Hd) = 








Figure 2: Deviations from the ideal state may be visualized as a random walk through a lattice of n-dimensions 
(here only two are shown). The number of paths of equal length by which one can return to the origin increases 
rapidly with the distance. For simplicity one may think of the axes as deviation due to policy and deviation due 


to usage. 
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This grows rapidly with the Euclidean distance ld\;: 


d= a-y/de 
i=! 


The conclusion is that it is in the system’s best 
interests to remain close to the ideal state at all times. 
If the remedy to a particular large deviation were 
unknown, the search for a remedy, in state space, 
would become extremely time-consuming as the mag- 
nitude of the problem increased. The expense or 


‘hopelessness’ H(d) measures this more than exponen- 
tially divergent problem. This hopeless search is the 
fate of any immune system without specific expert 
knowledge. 


To summarize, every computer system, with a 
system policy, has an ideal state which is based on 
policy and environmental considerations. This ideal 
state fluctuates and degrades with time. The aim of 
system administration is to regulate the system to be 
close to this ideal state. 


Modelling Computer Behaviour 


An analysis of behaviour in a computer system, 
interacting with its environment of users and network 
impulses, requires two types of model, which may be 
referred to as passive (type I) and strategic (type IJ). 
The distinction refers to the level perceived intent 
behind the changes which take place. 


In a type I description, the computer is viewed as 
being a mechanism coupled to a pseudo-periodic, ran- 
dom bath of impulses from an environment. One con- 
siders the effect of the this signal of impulses on the 
state S(t). This is the view taken in [7, 8, 16, 4]. A type 
I model relates the behaviour of computers to that of 
other interacting, dynamical systems in mathematics 
and physics. This type of description is easily formu- 
lated and can be used to predict some of the average 
behaviour when the system is approximately stable. It 
works particularly well when large numbers of users, 
or transactions are involved, butnotvery well in situa- 
tions of low usage. It is not good at predicting signifi- 
cant change, however, since the assumption of the 
method is that only gradual changes takes place over 
relatively long periods of time. 


In a type II description, the computer system is 
viewed as the chequerboard for a game of competition 
between motivated individuals. This is the view taken 
in [3]. This type of analysis is designed to analyze the 
competitive processes which instigate significant 
change, at a more detailed level. It 1s good at deter- 
mining the probability of success when using a set of 
strategies, and or finding the optimal strategies to 
solve a particular problem, but it is more difficult to 
apply than a type I description and relies on a know!l- 
edge or intuition of every relevant strategy which 
might be used by administrators and users alike. 


Influences on the system can thus be classified as 
either random, stochastic or passive (type I), or as 
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intentional, adversarial or strategic (type II), depend- 
ing on the significance of the change. This distinction 
is partly artificial: all changes can clearly be traced 
back to the actions of humans at some level, but it is 
not always useful to do this. Not all users act in 
response to an important provocation, or with a spe- 
cific aim in mind. It just happens that their actions 
lead to a general average degradation of the ideal 
state, no malice intended. Thus there is a part of the 
spectrum of changes which averages out to a kind of 
faceless background noise: the details of who did what 
are of no concern [7, 8]. 


From type I models, based on the empirical stud- 
les made at Oslo, we have found that computer sys- 
tems behave remarkably like photonic gases in the 
limit of long times [16]. That is, the occurrence of 
events on computer systems mimics the behaviour of 
black body radiation in physics, The interpretation of 
any dynamical variable as a fluctuating, statistical 
quantity is made possible by considering the effect of 
infinitesimal perturbation to dynamical variables q(t). 
One begins by defining averages and correlated prod- 
ucts of the fields g(t), with action S[q]. The action is a 
generating functional which determines the constraints 
on the behaviour of the dynamical variable g(t) by a 
variational principle. For the simplest dynamical sys- 
tems, one may write 


] a 
S= f dV, = qOq) 
The sum over all fluctuations of given latency may be 
written [4]: 
I[< g(t) >] = — In f du esha Fa: 
TB 


The subscript TB refers stands for ‘transaction 
bubbles’ and refers to correlation graphs which are 
closed loops, 1.e., complete transactions. This form is 
useful, since it is a self consistent form, which is 
derived on the assumption of linear statistical fluctua - 
tions an periodic time. It allows one to express self- 
consistent behaviour in terms of the measured vari- 
ables alone. This quantity is essentially the free 
energy; it is a sum over all complete transactions in a 
fluctuating system and relies only on the assumed 
microscopic model which specifies available freedom 
and applied constraints. It can be calculated and com- 
pared to the fluctuation distributions measured for sys- 
tem variables. 


Although the model is simple, the agreement 
with measured values is reasonable. The reason for 
this is a subtle but fascinating interaction between ran- 
domness and the order brought about by fixed daily 
and weekly rhythms. In fact, ensembles of events col- 
lected over weeks or months are insufficient in num- 
ber to be perfectly described by these statistical meth- 
ods, but the statistical model provides an idealized 
limit for computer behaviour, 1.e., 1t provides a well 
defined envelope which approximates the system at 
scale of weeks. Moreover, it is so much simpler to 
understand than the actual behaviour of the system, 
that it has a valuable role to play in the discussion. 
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These studies have shown that computers behave 
like co-cyclic oscillators with periods of one day and 
one week (the rhythms imposed by the environment of 
users). Over periods of time which are long enough to 
gather enough data, these cyclic constraints reveal 
themselves as the shapes of distributions of events 
over time. They offer predictions about the statistical 
nature of the signal. 


A type I model describes the average level of 
activity or state of the system which is related to the 
background noise. The second type of analysis which 
is required for a computer system is the analysis of 
non-cooperative user behaviour, 1.e., analyzing which 
aspects of user behaviour affect the distribution of 
resources in the system. This analysis must be based 
on the system policy, since cooperation implies that 
the system is operating either within or outside the 
bounds of behaviour implied by the policy. Analysis 
must attempt to evaluate objectively the efficacy of 
different work patterns (strategies) employed by users 
in their interaction with the system. 


A suitable framework for analyzing conflicts of 
interest, in a closed system, is the theory of games [17, 
18]. Game theory is about introducing players, with 
goals and aims, into a scheme of rules and then ana- 
lyzing how much a player can win, according to those 
restrictions. Each move in a game affords the player a 
characteristic value, often referred to as the ‘payoff.’ 
Game theory has been applied to warfare, to eco- 
nomics (commercial warfare) and many other situa- 
tions. In this case, the game takes place on the n- 


dimensional board, spanned by the d vectors. 


Resource management is a problem of eco- 
nomics, just as energy flows in physical systems are to 
do with the economics of energy. The difference in 
system administration is only that there is no a priori 
currency for describing the economics of system 
administration. It is necessary to invent one. In social 
and economical systems one has money as the book- 
keeping parameter for transactions. In physical sys- 
tems, one has energy as the book-keeping parameter. 
These quantities count resources, in some well-defined 
sense. 


There are several types or classifications of 
game. Some games are trivial: one-person games of 
chance, for example, are not analyzable in terms of 
strategies, since the actions of the player are irrelevant 
to the outcome. In a sense, these are related to the first 
kind of model referred to above. Some situations in 
system administration fit this scenario. More interest- 
ing, 1s the case in which the outcome of the game can 
be determined by a specific choice of strategy on the 
part of the players. The most basic model for such a 
game is that of a two-person zero-sum game, or a 
game in which there are two players, and where the 
losses of one player are the gains of the other. 


One feature which distinguishes the analysis pro- 
posed here from pure game theory is that the value 
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associated with different courses of action is not con- 
stant, but a function of time. The periodicities, dis- 
cussed in the previous section must be taken into 
account as well as longer term changes, finite limits of 
system resources, non-linearities and so on. The impli- 
cation of this 1s that the usefulness of a particular strat- 
egy varies according to when it is implemented. 


The first kind of analysis assumes that the sys- 
tem has an average state and can therefore be used (at 
least in principle) to detect anomalous behaviour, e.g., 
behaviour which contravenes system policy. The sec- 
ond type of analysis looks at specific behavioural 
traits and attempts to evaluate their implications for 
the system state in more detail. Whether user 
behaviour lies within or outside the bounds of system 
policy is a matter of choice. Presumably one is inter- 
ested in looking at all common behaviours, weighted 
by their likelihood in order to determine whether the 
system policy is effective enough. To make a type II 
theory realistic and tractable, one can imagine approx- 
imating the average background of the system activity 
using a type I model, and then studying specific strate- 
gies against this background. This leads to the notion 
of payoff, system currency in hybrid models. 


Payoff in Type I and Type II Hybrids 


Type I and type II models should not be should 
be thought of as completely separate issues: the best 
possible understanding of a computer system must 
involve both. Nonetheless it is primarily type II mod- 
els which offer the chance to evaluate procedures and 
strategies of system administration. Type I models 
provide the background understanding of the resource 
behaviour, required to give substance to a type II 
model. 


Equipped with a type I model for understanding 
the average interaction between user and system 
(which can be verified experimentally), one can con- 
struct a type II model in order to study a particular 
issue, against the average backdrop of type I activity. 
What is the outcome of introducing a new policy for 
governing a particular system resource, given what is 
known about how users generally interact with the 
system? 


The determination of payoff, or the currency of 
the game is the central problem now. In order to find 
strategies which can keep the system close to its ideal 
state, one must assign a realistic value to strategies 
employed by users and system administrators. This is 
done by formulating a matrix (table) whose rows and 
columns specify the value or payoff associated with 
particular courses of action, for one of the players (see 
Figure 3). In the zero-sum approximation, it does not 
matter which player is chosen, since the losses of the 
one are the gains of the other. This is the only case to 
be considered here. 


Courses of action available to each party, label 
the rows and columns. Rows are strategies and 
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columns are counter-strategies, or vice versa. The val- 
ues within the matrix are the values gained by one of 
the players, in units of the arbitrary currency of the 
game when a given row-strategy and column-strategy 
are chosen. These values are determined by policy and 
by information about how resources behave, acquired 
from type I models: they are a set of value judgements 
about what it important or unimportant in the system 
and to what degree. 


Counter-strategies 


Strategies 


/ 
Figure 3: The payoff matrix is a table of strategies 
and counter strategies. 


Once this ‘payoff’ matrix has been formulated, it 
contains information about the potential outcome of a 
game or scenano, using the strategies. This forms the 
basis for the theory of games [17, 18], whose methods 
and theorems make it possible to determine the opti- 
mal course or courses of action in order to maximize 
one’s winnings. Obviously, any and all information 
which contributes to a judgement is useful, however 
one does not necessarily need a particularly detailed or 
accurate description to begin making simple value 
judgements about system behaviour. Even a simple 
quantification is useful, if it can distinguish between 
two possible courses of action. 


How much can a user or an attacker hope to win? 
From our basic assumptions, the aim of a user is to 
maximize work produced or, in the worst case, maxi- 
mize resources consumed. The system administrator, 
or embodiment of system policy, is not interested in 
winning the game for resources in the same way as 
users, but rather in confounding the game for users 
who gain too much control. The system administrator 
plays a similar role to that of a police force. In a vague 
sense, the administrator’s jobs is to make sure that 
resources are distributed fairly, according to the poli- 
cies laid down for the computer society (a Robin Hood 
role of altruistic government). 


What is the currency of this evaluation? A defi- 
nition 1s required in order to quantify the production of 
useful work by the system and its users. Clearly the 
term ‘useful work’ spans a wide vanety of activities. 
Clearly work can increase and decrease (work can be 
lost through accidents), but this is not really germane 
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to the problem at hand. The work generated by a user 
(physical and mental work and then computationally 
assisted results) is a function of the information input 
into the system by the user. Since the amount of com- 
putation resulting from a single input might be infi- 
nite, in practice, the function is an unknown. 


In addition to the actual work produced by a 
user’s strategy, other things might be deemed to be of 
value, such as privilege and status. In a community, 
wealth does not guarantee privilege or status unless 
that coincides with the politics of the community. Pay- 
off can therefore be a complex issue to model. If one 
includes these ranking issues into calculations, one 
might allow for the possibility that a user plays the 
system rules in order to gain privileges for some later 
purpose. A user who accrues the goodwill of the sys- 
tem administrator, might eventually gain trust or even 
special privileges, such as extra disk space, access to 
restricted data etc. Such problems are of special inter- 
est in connection with security [19, 20]. 


For simplicity, the discussion of type II models 
in this paper refers only to games with two players. In 
a community, games are not necessarily two player 
zero sum engagements however. What is lost by one 
player is not necessarily gained by an obvious oppo- 
nent. Moreover, the information available to different 
sides in a conflict can affect their modes of play. The 
so-called prisoner’s dilemma, leads to the famous 
Nash equilibrium [21] which is a trade-off: 

A user of the system who pursues solely 

private interests, does not necessarily pro- 

mote the best interest of the community as 

a whole. 
Should users cooperate or fight to maximize their win- 
nings? Users can sabotage their own self-interest by 
using up all the available resources on a finite system, 
gaining enemies or losing the goodwill of system 
police. Strategies which succeed in encouraging users 
to comply with guidelines can therefore be an effec- 
tive way of ensuring a fair use of resources. The main 
reason for considering two person games here is the 
overriding simplicity of the two person game, com- 
pared to including more players. This should not be 
taken to imply that more complex models will not be 
important. 


In a realistic situation one expects both parties in 
the two-person game to use a mixture of strategies. 
The number of possible strategies is huge and the 
scope for strategic contrivance is almost infinite. 
Strategies can be broken down into linear combina- 
tions of primitives just as any operation on the system 
can. What then 1s a strategy? 

e An array of operations 
e A schedule for the operations 
e Rules for counter-moves or responses 


In addition to simple strategies, there can be 
meta-strategies, or long-term goals. For instance, a 
nominal community strategy might be to: 
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e Maximize productivity or generation of work. 
e Gain the largest feasible share of resources. 


An attack strategy might be to 
e Consume as many resources as possible. 
e Destroy key resources. 


Other strategies for attaining intermediate goals 
might include covert strategies such as bluffing 
(falsely naming files). Users can obey or ignore policy 
restrictions, use decoys, escalate or mitigate hostilities, 
attack/kill/delete a resource, retaliate. Defensive 
strategies might involve taking out an attacker, counter 
attacking, or evasion (concealment), exploitation, 
trickery, antagonization, incessant complaint (spam), 
revenge etc. Security and privilege, levels of access, 
integrity and trust must be woven into algebraic mea- 
sures for the pay-off. One of the advantages of this 
formulation on system administration is that it places 
regular administration on the same footing as security 
issues. These were never separate issues and should 
not be considered as such, even in today’s more secu- 
rity aware climate. 


A means of expressing these devices must be for- 
mulated within a language which can be understood 
by system administrators, but which is primitive 
enough to enable the problem to be analyzed alge- 
braically. 


Example Games 


The difficult part of a type II analysis is tuning 
the high level concepts and aims listed above, into 
precise numerical values. To illustrate the procedure, 
consider an example of some importance, namely the 
filling of user disks. The need for forced garbage col- 
lection has been argued on several occasions [22, 5, 
12], but the effectiveness of different strategies for 
avoiding disk may now be analyzed theoretically. This 
analysis is inspired by the user environment at Oslo 
University College, and the expressions derived here 
are designed to model this situation, not an arbitrary 
system. 


The currency of this game must first be agreed 
upon. What value will be transferred from one player 
to the other in play? There are three relevant measure- 
ments to take into account: (i) the amount of resources 
consumed by the attacker (or freed by the defender); 
sociological rewards: (11) ‘goodwill’ or (111) ‘privilege’ 
which are conferred as a result of ee to the 


Users/System 


Never tidy (2,1) _| 1) 


Conceal files 


Ask to tidy Tidy by date 


Tidy when asked = 1) 
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policy rules. These latter rewards can most easily be 
combined into an effective variable ‘satisfaction.’ A 
‘satisfaction’ measure is needed in order to set limits 
on individuals’ rewards for cheating, or balance the 
situation in which the system administrator prevents 
users from using any resources at all. This 1s clearly 
not a defensible use of the system, thus the system 
defenses should be penalized for restricting users too 
much. The characteristic matrix now has two contribu- 
tions, 
mz = 2,(resources) + 7,(satisfaction) . 


It is convenient to define 


1 /Resources won 
1, = m(resources) = = | ———__——_ 
Total resources 


Satisfaction 7, 1s aes arbitrarily on a scale from 
plus to minus one half, such that, Satisfaction 7, is 


assigned arbitrarily on a scale from plus to minus one 
half, 


IA 


i 


IA 


Nes. 


7 =) 

The pay-off is related to the movements made through 
the lattice d. The different strategies can now be 
regarded as duels, or games of timing; see Table 2. 
These elements of the characteristic matrix must now 
be filled, using a model and a policy. A general 
expression for the rate at which users produce files is 
approximated by: 

a Ngry + nor zg 

ny, + Ng 

where 7, is the rate at which bad users (1.e., problem 
users) produce files, and r, is the rate for good users. 
The total nE of users 1, =n, + ng. From experi- 


u 9 


ence, the ratio —2 is about one percent. ‘The rate can be 
n 


g 

expressed as a scaled number between zero and one, 

for convenience, so that r, = 1 — rz. 
The payoff in terms of the consumption of 

resources by users, to the users themselves, can then 

be modelled as a gradually accumulation of files, in 


daily waves, wnlen are a maximum around midday: 
5 fa r,(sin(27t/24) + 1) 
Ror 


where the factor of 24 is the human daily rhythm, 
measured in hours, and R,,, is the total amount of 


Tidy above | 
Threshold 
(1,3) (1,4) 
m(2,3) | (2,4) 


TN 2) 
(2,2) 





Change timestamps (4, 1) 7(4,2) (4,3) (4,4) 


Table 2: Games of timing. 
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resources to be consumed. Note that, by considering 
only good user or bad users, one has a corresponding 
expression for z, and ,, with r, replaced by rg or r, 
respectively. An automatic garbage collection system 
(cfengine) results in a negative pay-off to users, i.e., a 
pay-off to the system administrator. This may be writ- 


ten 
f : 
| ‘(sin(227/T,,) + 1 
fae ( ] ( p) ) 


ha-7 ’ 
2 Rrot 


where 7,, is the period of execution for the automatic 
system (in our case, cfengine). This is typically hourly 
or more often, so the frequency of the automatic cycle 
is some twenty times greater than that of the human 
cycle. The rate of resource-freeing r, is also greater 
than *,,, since file deletion takes little time compared 
to file creation, and also an automated system will be 
faster than a human. The quota payoff yields a fixed 
allocation of resources, which are assumed to be dis- 
tributed equally amongst users and thus each quota 
slice assumed to be unavailable to other users. The 
users are nonchalant, so xz, =0 here, but the quota 
yields 
l | 





nt,~=tt-— 
2 | ap + ry 
The matrix elements are expressed in terms of these. 
(1,1): Here 2, = — 1/2 since the system adminis- 


trator is as satisfied as possible by the users’ 
behaviour. 7, is the rate of file creation by good 
USES 2p, 1.e., Only legal files are produced. 
Comparing the strategies, it is clear that (1,1) 
= (1,2) = 2(1,3). 

(1,4): Here z, = 0 reflecting the users’ dissatisfac- 
tion with the quotas, but the system administra- 
tor is penalized for restricting the freedom of 
the users. With fixed quotas, users cannot gen- 
erate large temporary files. 2, is the fixed quota 
payoff, a fair slice of the resources, Clearly 
(4,1) = (4,2) = 2(4,3)= 2(4,4). The game 
has a fixed value if this strategy is adopted by 
system administrators. However, it does not 
mean that this is the best strategy, according to 
the rules of the game, since the system adminis- 
trator loses points for restrictive practices, 
which are not in the best interest of the organi- 
zation. This is yet to be determined. 

(2,1): Here z, = 1/2 since the system administrator 
is maximally dissatisfied with users’ refusal to 
tidy their files. The pay-off for users is also 
maximal in taking control of resources, since 
the system administrator does nothing to 
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prevent this, thus 7,=2,. Examining the 
strategies, one finds that 72(2,1) = (3,1) = 
(3,2) = (3,3) = 2(4,1) = 2(4,2). 

(2,2): Here 2, = 1/2 since the system administrator 
is maximally dissatisfied with users’ refusal to 
tidy their files. The pay-off for users is now 
mitigated by the action of the automatic system 
which works in competition, — thus 
1, =, — X,. The automatic system is invali- 
dated by user bluffing (file concealment). 

(2,3): Here 2, = 1/2 since the system administrator 
is maximally dissatisfied with users’ refusal to 
tidy their files. The pay-off for users is miti- 
gated by the automatic system, but this does not 
activate until some threshold time is reached, 
1.e., until f > fo. Since changing the date cannot 
conceal files from the automatic system, when 
they are tidied above threshold, we have 7(2,3) 
= 7(4,3). 

Thus, in summary, the characteristic matrix is given 

by Formula | where the step function is defined by, 


0 (t<% - 
and represents the time-delay in starting the automatic 
tidying system in the case of tidy-above-threshold. 
This was explained in more detail in [3]. 


0 -9=| 


It is possible to say several things about the rela- 
tive sizes of these contributions. The automatic system 
works at least as fast as any human so, by design, in 
this simple model we have 

= 2 [tal 2 lal = Wpigl 2 0, 
for all times. For short times 2, > 7, but users can 
quickly fill their quota and overtake this. In a zero- 
sum game, the automatic system can never tidy 
garbage faster than users can create it, so the first 
inequality is always saturated. From the nature of the 
cumulative pay-offis, we can also say that 


] ] 
G + 2,,) = (5 + 2, + 2,0(% —1t)) = 


(S + 2, + Xa) ’ 
and 
] ] 
[eae steal ee eee 

Applying these results to a modest strategy of auto- 
matic tidying, of garbage, referring to Figure 4, one 
sees that the automatic system can always match 
users’ moves. As drawn, the daily ripples of the auto- 


matic system are in phase with the users’ activity. This 
is not realistic, since tidying would normally be done 


AZ) —1/2 + 2,(t) SWZ (b) Ng 

ye W2+4,(t) 1/22+2,0+2,() 1/22+ 2,10 + 7,00 —) A, 
1/2 + 2,(t) 1/2 + 2,,(¢) 1/2 + ,(t) i 
1/2 + 2,(0 1/2 + 2,(4 1/2 + 2,(t) + 2,(0O(% — 4 A, 


Formula 1: Characteristic matrix. 
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at night when user activity 1s low, however such 
details need not concern us in this illustrative example. 


The policy created in setting up the rules of play 
for the game, penalizes the system administrator for 
employing strict quotas which restrict their activities. 
Even so, users do not gain much from this, because 
quotas are constant for all time. A quota is a severe 
handicap to users in the game, except for very short 
times before users reach their quota limits. Quotas 
could be considered cheating by the system adminis- 
trator, since they determine the final outcome even 
before play commences. There is no longer an adap- 
tive allocation of resources. Users cannot create tem- 
porary files which exceed these hard and fast quotas. 
An immunity-type model which allows fluctuations 1s 
a more resource efficient strategy in this respect, since 
it allows users to span all the available resources for 
short periods of time, without consuming them for 
ever. 


According to the minimax theorem, proven by 
John Von Neumann, any two-person zero-sum game 
has a solution, either in terms of a pair of optimal pure 
strategies or as a pair of optimal mixed strategies [17, 
18}. The solution is found as the balance between one 
player’s attempt to maximize his pay-off and the other 
player’s attempt to minimize the opponent’s result. In 
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general one can say of the pay-off matrix that 


maxminz,, S minmax Z,,. , 
i Te many MEL 


where the arrows refer to the directions of increasing 
rows (|) and columns (~). The left hand side is the 
least users can hope to win (or conversely the most 
that the system administrator can hope to keep) and 
the right is the most users can hope to win (or con- 
versely the least the system admin can hope to keep). 
If we have Equation 2, 


max minz,, = minmaxZ,, 
=> ge t 


Equation 2: Equality in the payoff matrix. 


it implies the existence of a pair of single, pure strate- 


gies (r*, c*) which are optimal for both players, 
regardless of what the other does. If the equality 1s not 
satisfied, then the minimax theorem tells us that there 
exist optimal mixtures of strategies, where each player 
selects at random from a number of pure strategies 
with a certain probability weight. 


The situation for our time-dependent example 
matrix 1s different for small ¢ and for large t. The dis- 
tinction depends on whether users have had time to 
exceed fixed quotas or not; thus ‘small ? refers to 
times when users are not impeded by the imposition of 
quotas. For small ¢, one has: 





60 80 100 


Figure 4: The absolute values of pay-off contributions as a function of time (in hours), For daily tidying 7, = 24. 
User numbers are set in the ratio (”,,n,) = (99, 1), based on rough ratios from the author’s College environment, 
1.€., One percent of users are considered mischievous. The filling rates are in the same ratio: 7;/R,., = 0. 99, 
ro/Rry = 0.01, r,/Rio, = 0.1. The flat dot-slashed line is |7,|, the quota pay-off. The lower wavy line is the cumu- 
lative pay-off resulting from good users, while the upper line represents the pay-off from bad users. The upper 
line doubles as the magnitude of the pay-off |r| = |z,,|, 1f we apply the restriction that an automatic system can 
never win back more than users have already taken. Without this restriction, || would be steeper. 
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The ordering of sizes in the above minimum vector 1s: 
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For the opponent's endeavours one has 
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This indicates that the equality in Equation 2 1s satis- 
fied and there exists at least one pair of pure strategies 
which 1s optimal for both players. In this case, the pair 
is for users to conceal files, regardless of how the sys- 
tem administrator tidies files (the sysadm’s strategies 
all contribute the same weight in Equation 2. Thus for 
small times, the users are always winning the game if 
one assumes that they are allowed to bluff by conceal- 
ment. If the possibility of concealment or bluffing is 
removed (perhaps through an improved technology), 
then the next best strategy is for users to bluff by 
changing the date, assuming that the tidying looks at 
the date. In that case, the best system administrator 
strategy is to tidy indiscriminately at threshold. 


For large times (when system resources are 
becoming or have become scarce), then the situation 
looks different. In this case one finds that 

ee min Tre) = min she Trg = Mg . 
In other words, the quota solution determines the out- 
come of the game for any user strategy. As already 
commented, this might be considered cheating or poor 
use of resources, at the very least. If one eliminates 
quotas from the game, then the results for small times 
hold also at large times. 


This simple example of system administration as 
a strategic game between users and administrators is 
only an illustration of the principles involved in build- 
ing a type I/II hybrid model. In spite of its simplicity, 
it is already clear that user bluffing and system quotas 
are strategies which are to be avoided in an efficient 
system. The value of ‘goodwill’ in curbing anti-social 
behaviour should not be underestimated. By following 
this basic plan, it should be possible to analyze more 
complex situations 1n future work. 


Future Work 


From the type I models studied at Oslo [4, 7], it 
appears that the most important characteristic of the 
average user behaviour is its periodicity: the average 
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state of computers has a daily period and a weekly 
period; these trace the social cycles of users all around 
the world. It is possible, as more is leamed, that more 
detailed characteristics will emerge which are general 
enough to be used in a type I/type II hybrid. The main 
promise of type I theories lies in the possibility of 
anomaly detection and self-analysis, leading to fault 
detection, intrusion detection and improvements in 
immune system technology at the user level (e.g., 
cfengine). However, it is also important to know, for 
strategic analysis, when the system is most loaded, 
must vulnerable and most available. 


Only one example of a type II theory has been 
examined here. What other issues might be studied by 
a type II model? The possibilities include strategies 
such as: consolidation versus distribution in system 
planning (where should resources be located?); dele- 
gation vs centralization; choosing many simple tools 
or a few powerful ones [23] (cost of leaming and sup- 
port, functionality, likelihood of bugs, results, rate of 
evolution of task and tools); the effect of system work 
ethics on productivity in a business (does the business 
spend most of its time working against itself or its 
competitors?); 1s the best strategy one which leads to 
stability or perfection? Mission critical systems and 
high security systems are obvious candidates for anal- 
ysis. Other resources uses: network share, processes, 
setting of permissions, placement of security etc. The 
possibilities are limited only by the imagination. The 
benefit of the type II model is in setting up a system- 
atic method for making impartial judgements about 
strategies for system management and system regula- 
tion. 


A common theme in all strategic studies, involv- 
ing complex competitive behaviour, is the so-called 
Red Queen scenario. This is about working hard to 
maintain the status quo; it is a reference to a scene 
from Alice Through The Looking Glass: 

“Well in our country,” said Alice, still pant- 

ing a little, “‘you’d generally get to some- 

where else — if you ran very fast for a long 

time as we 've been doing.” 

"A slow sort of country!” said the Queen. 

“Now here, you see, it takes all the running 

you can do, to keep in the same place. If 

you want to get somewhere else, you must 

run twice as fast as that!” 
This is also referred to as an arms race. In a true 
dynamical system, nothing stands still. Adapting one’s 
strategies to be optimal over time means a continual 
reappraisal of their efficacy. One has to be running all 
the time to keep up with the environment, continually 
adapting to change. The need for garbage collection is 
an example of this. 


In choosing strategies which walk the line 
between compliance and conflict, within a framework 
of rules (the prisoner’s dilemma), some studies have 
indicated that the best solutions are often cooperation 
at first, and then wit for tat after that, if cooperation 
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does not work: 1.e., try cooperation first to mitigate 
hostilities, and then send a message that one means 
business thereafter. 


Sketching out this recipe for analyzing system 
administration policies reveals some potent ideas, 
which have a merit quite independently of their ana- 
lytical value. The idea of using mixtures of strategies 
to most efficiently regulate a system, is so close to the 
ideas used in cfengine [5, 12] as to suggest that they 
could be adopted more even explicitly, and more 
dynamically. Rather than relying on batch operation, a 
policy engine like cfengine could be more dynamical 
in its responses to deviations from ideal state, and be 
able to set in motion a variety of parallel responses, 
which might extend its reach in dealing with more 
dynamical problems like network intrusions. It could 
also respond to more long-term trends in system usage 
and adapt its behaviour accordingly. Part of the moti- 
vation of this work was precisely to see what could be 
done to improve on cfengine [24]. Once refined, the 
approach in this paper will lead to improvements in 
cfengine, and improve the automation of host security. 


Summary 


The aim of this paper has been to propose a 
framework for analyzing models of system adminis- 
tration. Its main contention is that it is possible to see 
system administration as the effort to keep the system 
close to an ideal state, by introducing countermeasures 
in the face of competitive resource consumption. This 
is the formal basis which opens the way for objective 
analyses in the field. 


With a mathematical approach, it becomes easier 
to see through personal opinions and vested interests 
when assumptions and methods are clearly and rigor- 
ously appraised. However, one can only distinguish 
between those possibilities which are taken into 
account. That means that every relevant strategy, or 
alternative, has to be considered. This is the limitation 
of game theory. It is not possible to determine strate- 
gies without the creative input of experts, and a clearly 
described policy. 


Appealing only to a simple-minded analysis of 
disk filling, some straightforward conclusions are pos- 
sible: the use of quotas is an inefficient way of coun- 
teracting the effects of selfish users, when the whole 
community’s interests are taken into account. A quota 
strategy can never approach the same level of produc- 
tivity as one which is based on competitive counter- 
force. The optimal strategies for garbage collection are 
rather found to lie in the realm of the immunity model 
[6, 15]. However, it is a sobering thought that a persis- 
tent user, who is able to bluff the immune system into 
disregarding it, (like a cancer) will always win against 
the resource battle. The need for new technologies 
which can see through bluffs will be an ever present 
reality in the future. With the ability of encryption and 
compression systems to obscure file contents, this is a 
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contest which will not be easily won by system admin- 
istrators. 
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An Expectant Chat about Script Maturity 


Dr. Alva L. Couch — Tufts University 


ABSTRACT 


Using scripts to automate common administrative tasks 1s a ubiquitous practice. Powerful 
scripting languages and approaches support seemingly ‘efficient’ scripting practices that actually 
compromise the robustness of our scripts, as well as indirectly detracting from the stability and 
maturity of our support infrastructure. This is especially true for scripts that automate complex 
interactive processes using the scripting tools Expect or Chat. I present a formal methodology for 
the design and implementation of interactive scripting that, with a little more effort than writing a 
simple Expect script, produces scripts with substantially improved robustness and permanence. 
My scripting tool Babble interprets a detailed structural description of an interactive session as a 
script. Using this declarative, fourth-generation language, one can craft interactive scripts that are 
easier to perfect, inherently more robust, easier to maintain over time, and self-documenting. 


Introduction 


The amazing powers of current rapid prototyping 
languages strongly entice us to ease our burdens by 
writing simple scripts to automate repetitive adminis- 
trative tasks. But the nature of our profession also 
encourages us to cut corners on these scripts, writing 
in haste to satisfy often inadequately predefined needs. 
Our scripts are not subjected to rigorous software 
engineering process or testing. They are easy to write 
but almost completely undocumented and difficult for 
anyone but the author to understand and maintain. The 
process of script writing 1s evolutionary rather than 
planned, driven by expediency rather than coherent 
overall design. This accelerates the ‘software rot’ that 
is unavoidable in all software development processes 
[3]. 

But even when we employ the best accepted 
software engineering process, writing system adminis- 
tration automation scripts 1s actually more difficult 
than writing many other types of software. Adminis- 
trative scripts have strong couplings to their operating 
environment and make substantive changes to their 
environment as they execute. They are highly embed- 
ded [2] systems with complex preconditions and 
requirements for script success. An administrative 
script can be faced with any pre-existing conditions, 
be required to modify anything, and be required to 
produce most any result. And errors in a script exe- 
cuted with administrative powers can have dire 
results. 


We can most easily understand the perils in writ- 
ing scripts by considering the target system configura- 
tion as a collection of global variables. No one writes 
‘normal’ software using global variables anymore, 
because of the danger of creating code that makes 
undocumented and untraceable changes in unpre- 
dictable places. But we do the equivalent in writing 
privileged administrative scripts on a day-to-day basis. 


Scripts and Organizational Maturity 


Scripts are not simply passive tools that we can 
use or ignore on a whim. Once deployed, they have an 
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active role in determining the maturity of our service 
organization as defined in the System Administration 
Maturity Model, or SAMM [18], based upon the 
Capabilities Maturity Model of software engineering 
[7, 29]. One goal of SAMM 1s to encourage stable 
organizational structures in which particular staff 
members are interchangeable and replaceable on a 
moment’s notice. The scripts that we craft to ease our 
lives can violate this principle in a rather subtle way. 


Ad-hoc scripts often possess hidden usability 
constraints and behaviors only known to the author. If 
they work when we use them, fine; else we page the 
author, who repairs the damage thus inflicted. It is 
easy for the rest of us to relax into complacency as 
long as the author responds to pages in a timely man- 
ner. But regardless of the benevolence and good inten- 
tions of the author, using such a script compromises 
the maturity of the whole service organization, 
because the author becomes an irreplaceable compo- 
nent and service bottleneck instead of being inter- 
changeable with other staff. 


Because of this effect, some site managers (who 
shall remain nameless) prohibit ad-hoc scripting and 
automation, so that anything that cannot be automated 
by high-quality, well-documented, industrial-strength 
automation tools is done entirely by hand, avoiding 
scripting wherever possible. I take the controversial 
stance that this seemingly strange decision 1s justifi- 
able. By making this choice, all their staff remain 
interchangeable and replaceable on a moment’s notice, 
thus increasing their support organization’s stability 
and maturity, at the cost of reducing individual pro- 
ductivity. 

I formed this controversial opinion from direct 
experience. I am not an average script writer. I have 
written over 30,000 lines of Perl in the ten years I 
have known the language, to achieve many different 
ends. But I have also had a unique opportunity to 
observe the impact of my own scripts upon operations 
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in my absence. When I ‘retired’ two years ago from 
technical to managerial duties, I left all of my ‘clever’ 
administrative scripts in the hands of another highly 
qualified staff member. Little by little, over the course 
of two years, I had to make the administrative decision 
to ‘retire’ each of these scripts in order to make opera- 
tions more efficient. Most of them were not crafted 
well enough to outlast my direct involvement in using 
them, so that I became a bottleneck in my own opera- 
tions whenever they failed. The only exceptions were 
scripts I very heavily documented and widely dis- 
tributed, at great personal effort. 


Assessing Script Maturity 


While we can assess the quality of scripts using 
traditional software quality metrics [25], the relative 
importance of these metrics depends upon how script 
quality affects service organization maturity. In this 
context, a script exhibits high quality when its use 
does not depend upon specialized and esoteric know!l- 
edge, so that any properly trained and authorized staff 
member can utilize it with predictable and helpful 
results. While traditional quality factors such as docu- 
mentation, reliability, robustness, and maintainability 
remain important, the peculiar properties of the 
administrative environment in which we utilize these 
scripts suggest some new quality factors that are more 
relevant and focussed upon our mission: 

1. precondition awareness: does the script under- 
stand the conditions under which it will func- 
tion correctly? 

a detection: can the script detect condi- 
tions under which it will not function 
and avoid problems? 

b assurance: can the script change the sys- 
tem so that preconditions are satisfied? 

2. convergence: does repeating the script produce 
the same effect? 

a self-consistency: does repeating the 
script produce the same results? 

b non-intrusiveness: does the script avoid 
repeating unnecessary intrusive actions 
that can potentially disrupt services? 

3. postcondition awareness: does the script check 
upon what it should be accomplishing? 

a verification: does the script check 
whether it did what it intended to do? 

b validation: do script changes have 
appropriate external effects, as observed 
from another machine? 

4. atomicity: does each script do related things as 
a unit, so that there are no partial effects that 
produce service failures? 

a transaction control: is there a mecha- 
nism whereby the script can detect par- 
tial completion? 

b rollback: is there a mechanism whereby 
the script can back out of changes made 
in the case of a failure? 

These factors all concentrate upon assuring pre- 
dictable script behavior that leaves the affected system 
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in a predictable and hopefully usable state, regardless 
of the identity of the particular administrator using the 
script. 
Avoiding Scripting 

The simplest way to avoid quality pitfalls of 
scripting is to utilize an automation tool whose design 
exhibits the above quality factors. Cfengine [4, 5, 6] 
and its relatives provide pre-written configuration 
methods possessing convergent properties. All the 
administrator has to do is to describe what to accom- 
plish, and Cfengine will accomplish it in the least 
intrusive way. Cfengine is highly aware of required 
preconditions and elegantly deals with their absence. 
It fails predictably if it cannot accomplish its tasks. 
Although it does not provide transaction control, a 
user can craft this through careful configuration [12]. 
In effect, Cfengine provides most of the control one 
can get from a script, and assumes responsibility itself 
for the quality of its actions. 


Cfengine is one of many tools available for 
avoiding scripting. My own Slink [9] solves the same 
problem for symbolic link tree hierarchies, and also 
provides a library of ‘effective administrative abstrac- 
tions’ [10] with appropriate convergent properties for 
use in custom Perl scripts. Other file distribution 
methods that avoid or otherwise encapsulate scriptable 
actions include RPM (which supports scripts for cus- 
tom actions) [1], rdist [8], and my own distr [11]. In 
my opinion, whenever one can replace scripts with 
powerful, reliable, and well-documented management 
tools, one should. 


Unfortunately, there are many very common 
administrative problems that current high-quality tools 
do not address. While assuring appropriate contents 
for configuration files is relatively easy via file distri- 
bution (and interactive editing) approaches, control- 
ling processes and other dynamic elements is a much 
more difficult task that usually requires some kind of 
custom scripting. 


Short of avoiding scripts, we can better manage 
them and avoid writing too many. PIKT [22, 23] pro- 
vides portability mechanisms for scripts that allow one 
script to function in a heterogeneous environment 
through preprocessing. Last year, we discussed how 
the logic programming language Prolog [12] sub- 
sumes the function of PIKT and supports script con- 
vergence, preconditions, and atomicity perhaps better 
than most scripting languages. But we concluded that 
coding in Prolog has its own unique difficulties and is 
not for everyone. 


Then how should the mere mortals among us 
arrange to receive the benefits of scripting without the 
detriments? Providing scripts with the appropriate 
kinds of robustness is expensive in terms of coding 
labor, but utilizing naive scripts may be equally 
expensive in terms of administrative stability, because 
only people ‘in the know’ can deal with their deficien- 
cles. 
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Interactive Scripting 


To better understand the problems involved in 
scripting, I utilize a scripting example problem which, 
to my knowledge, presents almost all possible difficul- 
ties. Almost all network components have serial ‘con- 
soles’ from which commands can be issued, and begin 
their lives in a state that requires some kind of manual 
configuration via interactive console commands. 
Routers, switches, and network appliances have to be 
assigned Internet addresses and networking informa- 
tion before I can utilize the Simple Network Manage- 
ment Protocol (SNMP) to finish the job. Typical 
UNIX and Linux servers must be built from the con- 
sole before I can utilize automated methods to com- 
plete configuration. And if the network dies, then the 
only ‘sure’ method of interacting with potential cul- 
prits is still the trusty serial console. 


Automating interaction with console interfaces 
poses many problems above and beyond just knowing 
how to write scripts. A human performing administra- 
tive actions must read reams of documentation, under- 
stand the meanings of commands, and adapt to mes- 
sages from the console to determine future commands. 
A script trying to mimic these actions begins execu- 
tion unaware of the device’s current state, meanings of 
commands, or history of changes. It must discover 
these by parsing dialogs as it executes. 


The easy way to configure a device is through 
‘Invasive’ scripting that erases the whole device con- 
figuration and starts over each time. This gives the 
script complete initial knowledge of the state of the 
device by clearing all data before making changes, 
which in turn makes writing the script a simpler task. 
For example, to add a new user, one can erase all users 
and then create them all again, including the new ones, 
much to the dismay of people currently using the 
device! 


A ‘convergent’ script [12] changes the device 
from an unknown initial state to a desired one, without 
unnecessarily interrupting concurrent use of the 
device. The script must discover that state, compare it 
with what is desired, and craft a minimal set of actions 
that will accomplish needed changes. This process is 
much more complex than simple ‘invasive’ scripting, 
but much more desirable because it will not interrupt 
the function of correctly configured devices. Most 
devices support this kind of interaction rather poorly. 
In fact, 

Convergence is not a property of the device, or 

of its configuration, but of our ‘best practices’ 

in managing it while maintaining an appropri- 

ate level of service for others. 
This makes crafting convergent scripts both particu- 
larly difficult and particularly important, as they 
embody all aspects of human interaction, including 
device knowledge, configuration requirements, and 
management policy. 
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My Goals 


I began the work of this paper by looking for a 
better way to write Expect [19] or Chat scripts that 
will enable ‘convergent’ bootstrapping and adminis- 
tration of console-scriptable network nodes. I needed 
something like this in order to be able to reliably 
recover from errors made by students in building 
experimental networks. Left to herself, and given full 
reign over a network device, a student can unknow- 
ingly break SNMP (or other) management control 
over network devices, thus making ‘front door’ recov- 
ery techniques unreliable. 


My second motivation was to bring the process 
of scripting closer to the ‘best practices’ I already 
understand. The tightest coupling I can make is to 
relate automated scripts to the commands I would 
have to issue myself in order to accomplish the same 
task. I consider this a much tighter ‘semantic cou- 
pling’ than, say, SNMP requests to accomplish the 
same changes: SNMP requests look very different 
indeed from the administrative commands to which 
they correspond. 


The Lightwave ConsoleServer 3200 


My example application 1s to create a convergent 
script that will automatically maintain the configura- 
tion of a LightWave ConsoleServer 3200 [30]. This is 
a serial console switch that allows access to any one of 
up to 32 serial consoles from up to 16 simultaneous 
incoming telnet sessions. It 1s remarkably easy to con- 
figure, but configuration involves setting many param- 
eters, and these may only be set by hand using a serial 
command-line interface. There is no SNMP interface 
available and management functions are not network- 
accessible by any means. This device, once config- 
ured, also allows script access to all other consoles in 
my site, via telnet within a dedicated private 
(RFC1918) administrative subnet. 


I wish to use the ConsoleServer in college 
coursework in order to give students access to remote 
Linux consoles, so that they may practice configuring 
Linux systems that are physically located in a pro- 
tected location. This means that the configuration of 
the ConsoleServer will be changing frequently in 
order to allow new students access to the consoles. I 
already maintain databases of the students who should 
be given access to particular consoles. The trick is to 
craft a convergent mechanism by which I can assure 
that the appropriate students have access to appropri- 
ate machines, by adding and deleting accounts for par- 
ticular students as they rotate through lab exercises. 


Easy or Impossible? 


One might think that this project is easy until one 
understands the true complexity in the interactions. 
Let us consider the simple subtask of making sure the 
switch is accessible to the correct students, and that 
they possess appropriate privileges. To add a user, one 
participates in a dialog similar to this: 
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Leis 20070 1osin admin 

PLEASE ENTER PASSWORD **** 

sys admin>adduser 

Number of available user records: 196 
Number of users defined: 4 

Enter user Ad’ USER 1D. >< foo 
Enter case sensitive password 

| -PASSWORDS> eta 

Re-enter case sensitive password 
 EPASSWORDiw 7 ats 

O-17 | MAX CONCURRENT LOGINS: 1 > 1 
Allowed devices example: 

150. || DEVICES <0: 2: ae tui 

Allowed listen devices example: 
15.0 DEVICES. <0. >: 50 

Allow user to clear device buffer 
(Y/N) | YES > N 

Clear screen after a command 
(L/Ne || VES > -Y¥ 

Enter user id | USER ID > 

sys admin>logout 

hET32007 


(in this paper, long lines in examples are folded to fit 
within columns). With Chat, one can craft this spe- 
cific dialog, but employing variables in the dialog is 
awkward at best. With Expect, one can write a TCL 
[24] subroutine that performs this task, where all user 
inputs are variables. Then one can call the subroutine 
multiple times to add multiple users. 


This all seems relatively straightforward until we 
consider what can go wrong during the script. The 
user could have been created already, so that all we 
need to do is to modify settings and privileges. This 
requires executing a different command edituser. The 
answers to any of the above questions, as specified by 
our script user, could be inappropriate. In this case the 
device repeats the question: 

Allow user to clear device buffer 
(Y/N) | YES > No 

Allow user to clear device buffer 
CEN) 4) CES: > 


The device does not accept ‘No’, just ‘N’. Improper 
answers leave the device asking the same question 
over and over again, so that the script must issue a 
control-C to reset the interface after any script failure. 
We must include one ‘if’ statement in our script to 
catch each possible failure. 


Making the script ‘convergent’ requires much 
more work. We must teach the script how to gather 
data on existing users, and how to determine users that 
have not yet been added. Modifying users so that they 
have new privileges is a matter of reading each user’s 
profile, comparing it with desired data, and changing 
it if necessary. Each of these processes is more com- 
plex than the example above. The net result is that the 
‘convergent’ version of the script has an enormous 
number of possible execution paths, depending upon 
what goes wrong. This high ‘branching complexity’ 
[20] will cause the script to be very expensive to write, 
debug, and maintain. 
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If I write this script under pressure, I am obvi- 
ously not going to have the time to do this correctly 
and will miss some case. So to use my script, I have to 
watch for failures, correct the values of parameters, 
and run the script again, perhaps after cleaning up 
after what it did the previous time. If I become impa- 
tient enough, I will modify the script to better handle 
some of the failures, in order of annoyance to me. 


As I address the annoyances, I create another 
problem. As my script becomes increasingly clever, it 
is evolving with abandon, without functional bound- 
aries or documentation. As it grows, it becomes like- 
wise increasingly clever and increasingly unmaintain- 
able. Any time the device changes, the script fails and 
I am the only hope of repairing it. I am well on the 
way to owning an irreplaceably valuable script that 
renders its author irreplaceable as well. This is what 
we call ‘job security.’ 


It is for this reason that I used to consider the 
goal of creating ‘convergent’ interactive scripts (that 
apply minimally intrusive changes in order to assure 
device state) practically impossible. 


Jackson System Design 


In software engineering practice, one manages 
project complexity and avoids this kind of develop- 
mental ‘script rot’ by applying a formal design 
methodology that controls development in order to 
keep scripts both understandable and reusable. Fortu- 
nately, the methodology we need was well understood 
during the punched card era of computing, and only 
needs to be resurrected and reapplied! 


Jackson [15, 16] claimed that the way to properly 
design a program for processing punched card stacks 
is to link the structure of the program with the struc- 
ture of the stack that it processes. He created a simpli- 
fied structural model that replaces program flowcharts 
with ‘Jackson Diagrams’ that are the same for the pro- 
gram and its input. Each diagram depicts containment 
and sequence of inputs or program parts, utilizing 
nodes for parts and undirected edges for relationships, 
reading from top to bottom and left to right. The dia- 
gram: 

A 
Fae ox 
B.. 2C>: 2D 

represents a thing ‘A’ that consists of subparts ‘B’, ‘C’, 
and ‘D’ in that order. This thing can either be a stack 
of cards or the program that processes them in sequen- 
tial order. Loops and branches in the program are indi- 
cated by annotating the diagram. Repeated items dur- 
ing a loop are starred, and optional items are annotated 
with a ‘0’. 

For example, the diagram below refers to a deck 
of cards containing a structure A, which consists of 
structures B, C, and D, where B consists of multiple 
copies of E, and D might contain either J or K. 
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Jackson’s key idea was to interpret this diagram 
also as the structure of a program to process the cards: 
begin A; 
begin B; 
for (some cases) 
begin E; 
begin F; end F; 
begin G; end G; 
end E; 
end B; 
begin C; end C; 
begin D; 
if (some condition) 
begin J; end J; 
if (some condition) 
begin K; end K; 
end D; 
end A; 


The program must look something like this if the cards 
are structured the same way. 
Applying Jackson’s Principle 
I began this project ‘expecting’ to utilize an 

enhanced version of Expect to address state awareness 
and convergence problems in traditional scripts. I ini- 
tially added enhanced handling of parsing and variable 
binding, similar to that in PIKT, in an effort to make 
scripts shorter and easier to understand. My approach 
to this problem changed dramatically in mid-project, 
however, when I realized that Jackson’s methodology 
applies to the structure of the input/output streams 
with which we as administrators control the device. 
The structure of these streams (of prompts and com- 
mands) is predetermined by device design and one’s 
intent as an operator. This pattern can be mimicked by 
a script in order to accomplish the same intent: 

The structure of a fully functional interac- 

tive script is exactly parallel to the branch- 

ing and looping structure of the device 

interactions in which it must engage. 


This observation would have come to naught if I 
had used Jackson’s diagrams as above, for they 
become unwieldy when used to describe interactions 
of this complexity. Fortunately, I did not need to uti- 
lize these, because one can use a variant of the Exten- 
sible Markup Language (XML) [13, 26] to perform 
the same function. For this, I employ the XML tags: 

1. <repeat>: the equivalent of Jackson’s star; indji- 
cates repetition of pattems within an I/O 
stream. 

2. <branch>: the equivalent of Jackson’s ‘0’; indi- 
cates that something is one of many possible 
options. 

For example, the Jackson diagram above can be repre- 
sented as: 
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<A> 

<B><E> 
<repeat><F/><G/></repeat> 

</E></B> 

<Gi 

<D> 
<junction> 
<bpranch><J/></branch> 
<branch></branch> 
</ june tion? 
€junction> 
<branch><K/></branch> 
<branch></branch> 
</jumetaon> 

</D> 

</A> 


where 
e <X> marks the beginning of X. 
e </X> marks the end of X. 
e <X/> marks the beginning and end of X. This is 
equivalent with <X></X>. 


With this model in mind, I realized that my so- 
called ‘high-quality scripts’ looked much more like 
structural declarations than scripts, and that the non- 
structural imperative commands were obfuscating my 
understanding of the structure that the scripts docu- 
mented. I was worrying about ‘which variables to set’ 
in the scripts, while I should have been worrying 
about the structure of interactions. 


My obvious next step was to split each script into 
two parts: one part that documents structure and 
another that acts upon that structure. In the beginning, 
I considered the ability to separate structure from 
action as an extra ‘toy’ capability of my scripting tool. 
But eventually, as my structural markup language 
evolved in its ability to express detailed structure, I 
realized that: 

For most purposes, crafting of individual 
interactive scripts can be replaced by an 
intelligent scripting engine that parses 
detailed structural specifications of the 
interface and its desired configuration, and 
then proceeds to assure that configuration 
by exploiting documented interface struc- 
ture. 


At first glance this method of scripting may seem 
ridiculously awkward, but in practice it is much easier, 
faster, and more reliable than scripting. Once the 
scripting engine is written, all one must do is to docu- 
ment the streams that it controls. This involves speci- 
fying the sequence, variant content, conditional struc- 
ture, and topology of the commands that the interface 
understands. One accomplishes this by collecting and 
annotating example sessions. If one can do something 
manually, and react to all possible responses, one can 
script the process. The ‘script’ becomes documenta- 
tion describing what varies in the sessions, what 
options or branches there are in the process, and the 
‘causal intent’ of each interaction, e.g., creating a user. 
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Stream-structured Design 


My variant of Jackson’s method documents the 
structure of I/O streams through a series of simple and 
straightforward steps. Each step requires modifying an 
example script of a user session by adding XML-like 
markup tags. These tags form a Stream-Structure 
Markup Language, or SSML. When this process is 
completed, I feed the sum total of all sessions I have 
recorded, all appropriately marked and annotated, to a 
‘scripting engine’ that I call ‘Babble’. This engine uti- 
lizes structural documentation to decide how to inter- 
act with the device. 


For example, let us first consider how one docu- 
ments the process that deletes a user from the Light- 
wave 3200. The first step is to collect an example dia- 
log that accomplishes this, using the script and cu com- 
mands upon a connected UNIX host. This produces a 
file: 


Script started on Sat -Sep 16: 162292132000 


% cu -l cua/b -b 8*M7™M 
Connected” G°M 


LCI3200>login admin*“M 

PLEASE ENTER PASSWORD *****M 

sys admin>deletu“H “Heuser foo 7M 
Delete user:foo Yes or No (N):Y"M 

sys admin>logoutM 

EC13200>" 20M 

Disconnected*“G"M 

% exit™M 

Script done. on: Sat. .sep. 16> 16230229) 2000 
where the prefix ~ 
ters. 


indicates invisible control charac- 


Standardize Input 


My second step is to remove the header and 
trailer, together with chaff such as °H that indicates a 
backspace (together with characters I backspaced 
over). I convert &, <, and > to their XML equivalents 
&amp;, &lt;, and &gt; so that they will not conflict with 
the XML tags I will add in the next step. I convert the 
remaining special characters to their Perl escape-string 
equivalents for easy readability and editing. 

\ro\n 

LOI3200&e¢; login\sadmin\r\n 
PLEASE\sENTER\ sPASSWORD\s****\r\n 
sys\sadmin&gt;deleteuser\sfoo\r\n 


Delete\suser:foo\sYes\sor\sNo\s(N):Y\r\n 


sys\sadmin&gt;logout\r\n 
LCI3200&gt; 


In the above, \r represents return and \n represents line- 
feed, \s represents space, and \007 represents bell (con- 
trol-G). After this transformation, whitespace is 
ignored in all further steps and may be used to indent 
for clarity. 


Mark Input and Output 
The next step is to annotate the remaining text so 
that I know what 1s input and what is output. There is 


not yet an automated way to do this, so I manually 
insert get and put tags to distinguish things the 
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interface sent from those I typed. At the end of this, 
untagged text will be ignored. 


<brook name="delete"> 
Cput? ¥rX/ pur 
<get>LC13200&gt:</get> 
<put>login\sadmin\r</put?>\n 
<get>PLEASE\sENTER\sPASSWOR\s</get> 
Cpu Ye NEC, put \n 
<get>sys\sadmin&gt:</get> 
<put>deleteuser\sfoo\r</put>\n 
<get>Delete\suser:foo\sYes 
\sor\sNo\s(N):</get> 

<put?Y\r</put>\n 
<get>sys\sadmin&gt;:</get> 

<put> logout \r</ put? \n 
<get>LC13200&gt;</get> 

</brook> 


I call a little part of an I/O stream a brook ('). One 
subtlety is that since the \r’s are typed by us but the \n’s 
are typed by the responding system, the \n are outside 
the respective put’s. If I instead place the \n inside the 
put, the scripting engine will add an extra line-feed to 
every command, perhaps with problematic results. 


Identify Variants 


The next step is to document which strings vary 
in the stream, depending upon what I wish to accom- 
plish with this script. I call these strings variants to 
distinguish them from traditional variables, with 
which they share only a superficial resemblance. In 
my example, only the administrative password and the 
name of the user to delete may vary. All else is 
always the same. I mark and name variants where 
appropriate, inserting (ignored) line breaks and inden- 
tation for readability: 


<brook name="delete"> 

<puty \r</ puto in 
<pet>LC13200&gt;</get> 
<put>login\sadmin\r</put>\n 
<get>PLEASE\sENTER\sPASSWORD\s</get> 
<put2 

<Var name="adminpass > * ***</ var >\r 

C/ put? \n 

<get>sys\sadmin] &gt:</get> 
<put>deleteuser\s 

<var name="username">foo</var>\r 

G/ put Ve 

<get>Delete\suser: 

<var pattern=" [a-zA-Z0-9]+">foo</var> 
\sYes\sor\sNo\s(N) :</get> 
<putoY\eGfpure Vn 
<get>sys\sadmin&gt;</get> 
<put>logout\r</put>\n 
<get>LC13200&gt;</get> 
</brook> 


There are two kinds of variants. A named variant 1s 
something to be placed into put commands or discov- 
ered during get commands. There are two of these: 
adminpass and username. A named variant has to be 
assigned a value in order to be put, but acquires a 
value after a get. An unnamed variant is only valid 
within a get and represents variant input to be matched 
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and skipped over, documented by a Perl regular 
expression pattern. 


In Expect, at this point, I would have to assign 
these values to variables outside the realm of the 
device language, in TCL. In SSML, these variable 
bindings are accomplished by comparing this descrip- 
tion(of where variables appear) with a different XML 
database of appropriate values for each variant. In this 
way specifics of configuration are kept separate from 
the process by which one configures a thing. 


Classify Echo Types 


The next step is to carefully classify variant out- 
put into one of several classes. There are three classes 
of output, corresponding to echo options: normal echo 
(full duplex, the default), no echo, and starred echo 
(for passwords). These are indicated by tags that con- 
tain output within a put: 

<put?Cstars? 
yar nane="adminpass!' >" * *</ var> 


K/ stad so VES puro vn 


The default is that what I type shows up in the 
output in full-duplex mode. Placing output inside a 
stars tag documents that stars are displayed instead of 
what I type, while a noecho environment indicates that 
there is no echo at all, as in typical password dialogs. 


Document Conditional Behavior 


The next step is to indicate any branching or con- 
ditional behavior in the overall flow of the script. First 
it is possible that I will already be logged in when the 
script starts. In this case, I wish to skip the administra- 
tor login, a simple branch: 

Court Vr pit \n 
Cjunetion? 
<branch> 
<get2LCr3200bet:</ eet? 
<put>login\sadmin\r</put>\n 
<get>PLEASE\sENTER\sPASSWORD \s</get> 
Cput?<stars? 
var name="adminpass">** *"</ var? 
</ stars 2 \r</ put? Vn 
<get>sys\sadmin&gt;</get> 
</branch> 
<branch> 
<get>sys\sadmin&gt ;</get> 
</branch> 
</junction> 
Each branch starts with a get that is used to select 
which branch to execute. If 1 receive a non-administra- 
tive prompt, I log in, else I skip the process. Appear- 
ances can be deceptive: this is not a method, but docu- 
mentation. The stream can take two paths, and I have 
now documented both of them. 


Another branch will be taken if the user name I 
choose does not exist. I can document this branch by 
collecting more data: 

sys admin>deleteuser foo 
User foo does not exist 


To deal with this case, which is perfectly acceptable 
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since I wanted to delete the user anyway, I can modify 
the master script by adding a branch describing the 
new response: 


<put>deleteuser\s 
<var name="username">foo</var>\r 
</put>\n 
junction? 
“branch? 
<get>Delete\suser: 


<var pattern="[a-zA-Z0-9]+">foo</var> 
\sYes\sor\sNo\s(N): 
</get> 
Couto ES puto An 
</ branch? 
<branch> 
<get>User\s 
<var pattern="[a-zA-Z0-9]+">foo0</var> 
Nedoes\snot\sexist(/ ger? 
</ eet? 
</branch> 
</junction> 
<get> 
sys admin&gt; 
</get> 
If I receive the error message, I simply ignore it. The 
default action in SSML, if there is no match to a 
branch, is to fail with a script error. 


Associating Values With Variants 


The next step is to actually invoke a script engine 
upon the documentation in order to perform the docu- 
mented function. This is a matter of binding variants 
to appropriate strings and calling the scripting engine 
to interpret the results. This in turn requires creating 
declarations of variants separate from — but in agree- 
ment with — the stream declarations I have made. For 
example, for my brook described above, I might 
declare: 

<var name="adminpass">PASS</var> 
<var name="username">couch</var> 


to assign values of PASS and couch to adminpass and 
username, respectively. For simplicity, variant values 
are organized in a single global declaration in a sepa- 
rate file. 


Repeating Commands 


Most of the time, however, I do not wish to 
delete just one user. In SSML, I accomplish repeated 
tasks implicitly (as we did previously in Prolog [12]) 
by declaring sets of instances of variable values to use. 
I force an action to repeat by defining a variant that 
holds a set of instances, matched with a repeat markup 
that processes the instances. The structure of instances 
in the confi guration data must be parallel to the struc- 
ture of repeat tags in the markup. This process is aided 
by a simple but powerful name scoping mechanism. 


For example, suppose that I wish to delete both 
users foo and bar. I create a brook to do both, by insert- 
ing the brook I have already created into a repeat con- 
text: 
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<brook name="expunge"> 

<repeat instances="people"> 
<insert brook="removeuser"/> 
</repeat> 

</brook> 


The variant people consists of two instances of data 
needed by removeuser: 


<repeat name="people"> 

<instance> 
<var name="username">foo</var> 
<var name="adminpass">PASS</var> 
“fins tance> 

<instance> 
<var name="username">bar</var> 
<var name="adminpass">PASS</var> 
</instance> 

</repeat> 


Each set of distinct values is called an instance of the 
process. During the repeat, each instance is processed 
in turn. During processing of a particular instance, the 
script engine augments the top-level variant declara- 
tions with new variable values for each case in turn, 
then invoking the brook with these new values. Vari- 
ants declared outside the repeat clause keep their val- 
ues unless shadowed by definitions inside an instance, 
so I could have accomplished the same effect through: 
<var name="adminpass">PASS</var> 
<repeat name="people"> 
<instance> 
<var name="username">foo</var> 
</instance> 
<instance> 
<var name="username">bar</var> 
</instance> 
</repeat> 


As the variant adminpass occurs outside the block of 
instances, and is not shadowed within them, it is avail- 
able to the contents of the repeat markup for each 
instance. Variant bindings during a repeat are strongly 
typed but dynamically scoped. The kind of variant 
(repeat or text) must exactly match its usage, but vari- 
ants brought into scope by a repeat are available to 
any contained repeats or subprocess invocations. This 
allows one to declare multiply-dimensioned loops by 
defining two sets of instances with non-overlapping 
variant names, one set for each repeat. 


Discovering Configuration 


This variant binding scheme also works in 
reverse to allow us to inductively discover the values 
of variants for a set of instances. Suppose I want to 
get a list of all users. I know that the command for that 
is listusers: 

sys admin>listusers 
User id > COUCH 
User-ad > £00 

User id > BAR 

sys admin 


In my last use of repeat, the instances over which I 
iterated were all arguments to put commands. I can 
discover users by reversing this process, referring to 
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the variants within a get command: 
<brook name="listusers"> 
Cputp listusers\r</ put> in 
<while instances="people"> 
<get>User\sid\s>\s 
<var name="username" 
pattern="(A-20-9]+"> 
COUCH</var>\r\n 
</get> 
</while> 
<get>sys\sadmin></get> 
</brook> 


This creates several instances, all known under the 
name people, where each one contains the usemame of 
one user of the device. 


The difference between repeat and while lies in 
their control over instances. repeat does something for 
a fixed number of instances, while while inductively 
discovers instances as they appear in the output, and 
creates a list of all of them. This has the effect of 
updating variant space for the discovered values, eras- 
ing any previous value of the structured variant people, 
where each instance contains a current uSemame, 
mined out of the output by using the regular expres- 
sion pattern expression pattem “[A-Z0-9]+’. 

In this example, the while exits upon a timeout, 
after which instances discovered during each com- 
pleted pass through the while process become instances 
of the repeat variant people, overwriting any previous 
values. At this time we know all the names of users, 
and could now, e.g., delete all of them by invoking the 
‘expunge’ brook above. 


Convergent Processes 


Until the last example, I have not employed vari- 
ants that are computed at runtime. The reader might 
ask what good it does to discover variable values if 
there is no script to utilize these values. The scripting 
engine itself, with appropriate guidance, can utilize 
this data to great advantage, or even print a new con- 
figuration file representing the current configuration 
of the device. 


Assuring values of individual configuration 
parameters non-intrusively is fairly trivial. The engine 
reads them, and if they are incorrect, changes them 
accordingly. For it to do this, it is sufficient to instruct 
the engine on how to read and write particular config- 
uration parameters, with appropriate branching to deal 
with different cases. 


Difficulties arise, however, when one wishes to 
efficiently update a part of the configuration contain- 
ing an unknown number of instances of a thing. For 
example, in assuring that my idea of current users 
agrees with that of the device, I start with two lists of 
users, one in hand and one already configured on the 
device. To update the users, the engine must read cur- 
rent user information from the device, note differences 
between current and desired users, and proceed to 
modify the configuration so that the desired informa- 
tion becomes current. To empower the engine to 
behave intelligently in this case, I must document a 
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few processes that act on the same variants, including 
how to read the user list, how to read details on one 
user, and how to add, delete, and modify one user’s 
data. All of this information together describes a con- 
vergent process that will update the user list to have 
desired contents. 


So far, I have used SSML to describe more or 
less traditional program flow that is also representable 
using Expect. In this example Babble transcends 
Expect’s capabilities by responding intelligently to 
documentation. Babble reads the user list and adds, 
deletes, or modifies users as needed, checking its work 
at every step by reading what it has written. In this 
way, five short declarations are used to synthesize one 
incredibly complicated action that would be impracti- 
cal to code as a single declaration. This complexity 
thus migrates from the documentation into the script 
engine where it belongs. 


Exceptions 


Some devices have particularly annoying user 
interfaces. For example, the 3Com Corebuilder 9000 
prints the contents of SNMP traps on the console 
while one is trying to configure it, sometimes in the 
middle of typing commands. To configure this device, 
one must ignore these alerts while issuing configura- 
tion commands. One can do this in SSML by declar- 
ing an ‘exception’ pattem to check for and discard if 
present. This arranges for the trap data to be discarded 
whenever it appears, regardless of context. This would 
be incredibly awkward to arrange in Expect, as the 
exception pattern would have to be included in every 
pattern match in the whole script! 


Babble 


Babble is a scripting engine that parses SSML 
specifications and performs desired configuration 
tasks. It is implemented as a set of cooperating Perl 
packages that parse both stream documentation and 
variant declarations, and allow one to selectively 
invoke individual brooks or convergent processes with 
desired parameters. It is implemented as a Per] library 
because of the many and varied forms in which I store 
configuration information, in the hope that any exter- 
nal specification of configuration policy can be trans- 
lated into an appropriate set of variant declarations 
using Per]. Input to each invocation consists of an I/O 
stream with which to interact, a compiled SSML parse 
tree, a name of a branch to invoke, and a multi-level 
associative array describing variants and sets of 
instances to be processed. The output of each call is 
the modified variant array, modified to reflect any gets 
executed during the script. 


This version of Babble is so new that the only 
application I have so far crafted is the one described 
herein. I can report from this that crafting Babble 
scripts to control the 3200’s configuration was accom- 
plished unbelievably quickly, because scripts almost 
always worked correctly the moment they had correct 
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syntax according to the parser and builtin configura- 
tion tester included with Babble! Debugging was 
almost entirely a process of responding to complaints 
from Babble itself about mismatches of names, syntax 
errors, etc. The only specialized knowledge I needed 
was an understanding of how to accomplish specific 
things in Babble that I was used to accomplishing 
using scripting. 

While I designed Babble specifically for serial 
console interaction, it can be used to automate any 
serial interactive process, including UNIX commands, 
telnet, etc, in the same manner as Expect. E.g., one 
can use it to parse the output of ps and then use the 
result to interact with the process table using explicit 
kill commands. Babble does not — and never will — 
have the ability to perform direct system calls. 
Employing solely console commands documents ‘best 
practice’ at the expense of script speed and resource 
efficiency, perhaps a proper decision. 


Limitations 


Babble of course refers to the tower of Babel 
from the Bible, because like the builders of the tower, 
it speaks all languages, but without comprehension. 
This lack of comprehension 1s the root of all of its per- 
manent limitations. The commands Babble executes 
have no meaning to Babble itself, but are simply 
abstract patterns of interaction leamed from experi- 
ence. When that experience 1s somehow incomplete, it 
fails. When interacting with a complex device, this 
experience can never be complete and some failure is 
assured. 


State Coherence 


Babble’s greatest weakness is nearly invisible in 
the example. Every script in the example requires hid- 
den preconditions in order to function, and scripts 
must be strung together so that the postconditions of 
each script satisfy the preconditions of the next. The 
most important precondition is interface state, which 
is not even representable in the current markup lan- 
guage. The state of the interface indicates, e.g., 
whether the interface is in unprivileged or privileged 
mode, and whether the process is at a command 
prompt or within an interactive dialog. These hidden 
preconditions and postconditions affect the success of 
every script, in particular, every subbrook of a conver- 
gent process must start from — and end within — the 
same exact interface State. 


Version Control 


Another serious deficiency of Babble is that it 
cannot explicitly encode version information to assist 
it in dealing with identical hardware devices running 
different software or firmware. Each device revision 
requires a completely independent Babble script. This 
deficiency, alas, is completely intentional. 


Babble’s ‘topological algebra’ of structural tags 
is designed for automated merging of brooks that rep- 
resent special cases of the same task on the same 
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device. This will be accomplished in the future via 
‘parallel tree walks’ through the descriptions to be 
combined, in which one combined description 
emerges with appropriate junctions and branches 
inserted. So far, all attempts I have made to combine 
this feature with version control have compromised 
this automated merging capability, by making the 
algorithms for automated merging unnecessarily awk- 
ward or perhaps even impossible. 


Branching in Babble is a temporal phenomenon 
while versioning is spatial in character. Problems arise 
in temporal merging when spatial merging has been 
done first; one does not have enough information to 
complete the merge unless the streams being merged 
have identical spatial structure. I consider automated 
merging and temporal coherence more important than 
version control, and believe that version control may 
have to be handled by a completely different tool, 
much as PIKT provides a metalayer for managing ver- 
sions of normal shell scripts. 


Paranoia 


Babble’s run-time checking borders on paranoia. 
Unlike Expect scripts, which check only for specific 
cues in the input stream, Babble checks for full-duplex 
echo of output, as well as compliance of input with all 
markups one specifies. Scripts abort on any deviation. 
Babble also frequently ‘checks its work’ by reading 
parameters it has modified. 


If one wishes even more paranoia, Babble allows 
one to craft scripts that validate behavior rather than 
merely verify. Unlike verification, which simply 
checks that parameters are being set correctly, valida- 
tion checks that parameters have the appropriate exter- 
nal effects. For example, after enabling telnet on a 
device, one can telnet into it to check that it works; 
after creating a user, one can login as that user from 
another device. These detailed sanity checks would be 
impractical to craft via traditional methods, but are rel- 
atively easy to craft in SSML because the engine is 
handling most of the details of error detection and 
branching. 


Awkwardnesses 


Alas, Babble’s documentation format was driven 
by many expediencies. I used XML syntax because I 
was used to writing XML parsers. This was not the 
optimal match. Because of this, I had to escape all 
special characters in an awkward way so that they 
would not interfere with XML parsing. In fact, the 
syntax is not, strictly speaking, fully XML compliant. 
To make it possible to drop into Perl during a script, I 
had to allow embedded Perl, but in true XML one 
would have to escape &, <, and > in Perl code, render- 
ing it unreadable. Thus I allow regular Perl and pre- 
process the documentation file, escaping all special 
characters in Perl scripts before parsing the result as 
XML. 


This awkwardness, however, gives me the ability 
to incorporate features into Babble that are difficu It to 
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code for any other base language. A Babble configura- 
tion 1s a ‘literate program’[17] that represents several 
different facets of a process, including documentation, 
procedure, and policy, in one convenient package. The 
reason I chose XML for the base language of Babble 
was to enable me to render descriptions of these facets 
in HTML for viewing on any web browser. This is an 
invaluable debugging aid that will become a feature of 
Babble in the very near future. More important, the 
interactions that Babble undertakes with a device can 
themselves be represented in XML, so that each invo- 
cation can be described in HTML as well, with hyper- 
links from the transcript of the invocation to parts of 
the configuration that determined its shape. 


Critique 


This approach is superior to writing scripts with 
Expect for several reasons. It avoids classical verifica- 
tion problems associated with script development, so 
that process refinement is much less dangerous than 
when writing real scripts. This strength is negated, 
however, if one employs regular programming as part 
of processing a stream. Babble also avoids one of the 
main difficulties in scripting with Expect: the need to 
craft complex regular expressions to parse input of 
irregular structure. 


Avoiding Verification Problems 


The greatest obstacle to using traditional script- 
ing is that verifying the correct behavior of scripts is 
difficult. I avoid some of the difficulties by crafting 
documentation of a pre-existing condition, not a com- 
puter program. During the time I am tuning and per- 
fecting documentation, the ‘script engine’ that utilizes 
the documentation remains unchanged. I thus simplify 
the problem of verifying my process into two prob- 
lems: verifying the script engine itself as a program, 
and verifying the documentation as a description of 
external behavior. The engine need only be verified 
once. Verifying accuracy of its subsequent uses only 
requires checking its input for accuracy. 


This fact makes refining a description much eas- 
ler than refining a true script. The form of documenta- 
tion 1s sufficiently simple that traditional limits to 
script validation do not apply. One can use automated 
static verification tools to exhibit possible sub-paths, 
validate syntax, and check correspondence between 
stream and parameter declaration structure. Thus one 
can largely avoid the problems of ‘software rot’ that 
plague the maintainers of true scripts. 


The trick of separating documentation from pro- 
gramming only works if one can avoid embedding real 
program code into the documentation. If one must, 
one loses most of the benefits of the approach. In 
order to avoid this, one must be able to compile a 
complete parameter space before one starts the engine. 
If one cannot, but must compute configuration param- 
eters ‘on the fly’ during the script, simple process doc- 
umentation no longer suffices. Then one must drop 
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into Perl during a stream, so that my intended docu- 
mentation now again assumes the role of a program. 
Of course, when one must do this, one compromises 
many of the strengths of the approach, in the very 
same way that employing embedded TCL weakens the 
maintainability of Expect. 
Simplifying Regular Expression Syntax 

While regular expression pattern matching is one 
of the most powerful features of Expect, TCL, and 
Perl, it is also one of the most dangerous and 
unwieldy. The parenthetic syntax for binding sub- 
strings to output variables is a source of constant con- 
fusion. The most common error is counting parenthe- 
ses incorrectly so that variables are bound to incorrect 
values. In SSML, all parenthetic matching 1s implicit 
in the order of variables within a get, and parentheses 
are not enabled in the regular expression patterns. This 
makes the patterns much easier to craft and debug, and 
there is no danger of mismatched variables, as in par- 
enthetic patterns or split statements. 


This convention replaces one weakness with 
another (hopefully lesser) weakness. When crafting 
documentation, one must be careful to declare variants 
using appropriate regular expressions so that the pat- 
tern that the engine derives from your declarations is 
not ambiguous. For example, it 1s poor practice to 
declare two adjacent variants whose patterns create 
ambiguity in assigning values to the variants: 


<var name="poor" pattern="[(A-Z]+[0-9]+"> 


<var name="style"-pattern="(0-9)*"> 


When the engine tries to match these two variants 
against the input ‘AB1234’, the first pattern match 
causes an early binding of the variant poor to ‘AB1’, 
after which the second pattern matches ‘234’. But 
matching these patterns could just as well have split 
the input into AB12’ and ‘34’, ‘AB123’ and ‘4’, or 
‘AB1234’ and *’. 


Thinking Declaratively 


Several limitations of SSML are entirely inten- 
tional. One cannot negate a pattern in SSML, or use a 
regular ‘for’ loop. These are not simple oversights, but 
based upon fundamental limits of the theory of auto- 
matic program verification. 


Verifying the correctness of regular scripts with- 
out exhaustive testing is impractical. The most com- 
mon method of verification is called ‘weakest precon- 
dition analysis’ [14, 21, 28]. To use this method, one 
clearly documents preconditions and postconditions of 
the script, and then analyzes the script line by line 
from end to beginning, starting from desired postcon- 
ditions and carefully computing the preconditions 
needed to assure those postconditions. A script 1s ‘cor- 
rect’ if the preconditions actually required to assure 
postconditions are ‘weaker’ (1.e., less demanding) than 
the stated preconditions we document. This process is 
easy to perform automatically for scripts containing 
only linear code with no loops, but is equivalent in 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


An Expectant Chat about Script Maturity 


complexity to mathematical theorem proving for 
scripts containing loops in which the same variables 
are both set and used. Theorem proving takes far too 
much time to be practical. 


In practice, this means that the only practical 
way to assure the quality of a script is to put it through 
a full ‘regression test’ after any change. Because of 
the complexity of the environment in which adminis- 
trative scripts must run, this kind of testing 1s usually 
impractical or perhaps impossible. This allows inci- 
dental script bugs to remain hidden until they cause a 
crisis, perhaps until the original author has long ago 
moved on to other employment. 


To be able to efficiently verify a program, one 
has to ‘weaken’ the scripting language so that limits to 
automatic verification do not apply. SSML documen- 
tation contains no loops that would present problems 
for a ‘weakest precondition’ verifier, unless one inten- 
tionally reads a variable and sets it inside a repeat 
scope in the same brook. Doing this in SSML consti- 
tutes a markup error that future versions of Babble 
will be able to detect and report. 


The current implementation performs only lim- 
ited verification, including reporting disagreement on 
names and types of variants between stream documen- 
tation and value declarations. The structure of SSML 
will allow future versions of Babble to locate more 
kinds of common programming errors, including over- 
lap or ambiguity of regular expression patterns, as 
well as ambiguity of intent, such as writing data dur- 
ing a stream that should only be reading it, or vice 
versa. 


Relearning Common Techniques 


Writing SSML specifications requires that one 
learn new equivalents for common but less reliable 
scripting techniques. For example, it 1s very common 
for a traditional script to parse a line of input into an 
array with a split command. E.g., in Perl, one can 
write: 


Oparts = Splice Nest, oline): 


where /\st/ is a regular expression, $line is the unparsed 
line, and @parts is an array of fields within the line. 
Babble does not allow this kind of matching in a 
straightfor ward way, but there are two equivalent con- 
structions. First, one can name all parts of the line to 
be matched, and match them individually: 

Gar name="tirst' pattern—"[- s\n) "> 

<yar pattern="\st"> 

<var name="second" pattern="[7\n]+"> 

<var (pattern="\Vst"> 

varename> “third” pattern="(- ye\n| et? 


where the pattern [‘\s\n}+ matches non-whitespace, 
while the pattern \s+ matches whitespace. This will 
bind first, second, and third to the next three whites- 
pace-delimited fields. If there are a fixed number of 
fields, this is the best possible documentation on their 
structure. 
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If there are several fields for which one does not 
know a field count, such as a list of ports separated by 
commas, one can instead declare a repeating structure: 

<while instances="ports"> 
<junction> 
“branch> 

“yar name="port" 

pattern="[" \s\nt"> 
</var> 
</branch> 
<branch> 

<Var pattern="G\st"2C/var> 

(var “name> "port" 

pattern="[*\s\n]+"> 
</var> 
</branch> 
</junction> 
</while> 


The complexity here is more apparent than real. This 
declares a sequence of input in which there are 
repeated instances of a port, where each pair of port 
numbers are separated by whitespace and a comma. 
The branching structure indicates that it is possible 
that there is whitespace in front of each instance. If, 
e.g., the input is ‘2, 5’, the data that this process binds 
to the variant ‘ports’ has the structure: 
<repeat name="ports"> 
<instance> 
(var name="port™>2</var> 
</instance> 
<instance> 
(var name="port"275</ var? 
</instance> 
</repeat> 


This data can in turn become the argument to a repeat 
markup if one wishes to do the same thing to each dis- 
covered port! 


Automation 


Many steps in this process can be automated or 
streamlined so that much less user input is required. 
For example, when capturing example sessions, a tool 
that would capture and correlate both input and output 
(using time stamps to determine relationships) could 
generate the direction and echo markups that I created 
by hand in the example. 


There are subtle semantic difficulties in automat- 
ing the task any further without human intervention. 
For example, it is not possible to reliably infer the 
positions of variant data — or the regular expressions 
that describe them — from a few examples. A person 
must mark these. But once input, output, and variants 
are distinguished, multiple example sessions exhibit- 
ing different branches for the same task can reliably be 
combined automatically by parsing them, fusing their 
parse trees, and then printing the result. A person must 
nonetheless identify which set of brooks all accom- 
plish the same task and should be fused. Likewise, 
after I tell the engine which scripts read and write 
data, the engine can automatically determine whether 
write operations worked or not, by reading the results 
and checking those against my intent. 
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Conclusions 


When I began this work, I was possessed by the 
traditional spirit that scripting can solve any problem, 
and that all I had to do was to make scripting easier. 
Even when applying the relatively declarative thinking 
required for logic programming, I retained the old 
script mentality and first tried to do ‘everything I 
could do with Perl’. This attitude was the result of 28 
years of conditioning, and it took a long while to ques- 
tion this thinking, and even longer to unleam old 
habits in order to discover ways of doing without this 
‘expressive power’. 


The quality of our work, as script writers, is con- 
trolled by fundamental theoretical limits known to 
Computer Science. Normal scripts are difficult if not 
impossible to validate and verify by any method short 
of exhaustive testing. The unique properties of the 
administrative environment make this testing impracti- 
cal, while our lack of knowledge of the complete 
effects of our actions hampers top-down thinking and 
design. Babble cannot violate any of these limits, but 
can carefully work around them. It discourages unpro- 
ductive practices and shifts responsibility for script 
quality — whenever possible — away from the script 
itself and into a reliable intermediary component that 
better interfaces desires with devices. 


My joumey has been a ‘tale of power’. Some- 
times apparent power 1s an illusion. This illusion can 
cost us much time and effort to avoid. It can sap 
strength from our infrastructure while superficially 
pleasing our egos. It can keep us from realizing its 
effects, ‘trapping us in a lifestyle’ of seeming opu- 
lence with an underlying and ternble cost. But the 
first step in avoiding a trap is knowing of its existence. 


We each seek personal empowerment in our own 
ways, weaving a fabric of practices and tools that 
gives us the stability and security we all crave as 
human beings. Intrinsically we all know what the real 
‘best practices’ are: those techniques that enhance our 
personal empowerment and security. We may be given 
these by a superior, or discover them ourselves via bit- 
ter experience, but the result is the same. 


If we can document these practices so that they 
will outlast our attention and presence, we empower 
others in the same way. Thus we can move beyond the 
‘network of trust’ to form a ‘network of empower- 
ment’ in which our community of administrators is 
much stronger than the sum of its parts. This goal 
requires putting the community above one’s self-inter- 
est, in order that the community become strong 
enough to protect us better than we can protect our- 
selves. It requires looking beyond ‘job security’, 
toward ‘mission security’. It requires acting fairly 
within the ‘social contract’ that irrevocably binds us 
with our organization in a pact of mutual protection 
and shared mission. 


Because true empowerment flows not from 
inside ourselves, nor from our technologies, but from 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Couch 


caring community carefully woven around shared pur- 
pose, vision, and dreams. 


Availability 


Babble will be available soon in alpha release 
from http://www.eecs.tufts.edu/“couch/babble. While 
it is written entirely in Perl 5 and should be portable to 
all UNIX systems, the current version does not func- 
tion properly in Linux due to a bug in the CPAN 
pseudo-tty module Ptty.pm — I am working on this. 
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An Improved Approach for Generating 
Configuration Files from a Database 


Jon Finke — Rensselaer Polytechnic Institute 


ABSTRACT 


Much of our site configuration information 1s stored in a relational database, which means we 
need to extract this information in the appropriate format for servers and daemons. In the past we 
have done this with lots of little custom C programs and scripts. We have recently changed to a 
new approach of generating the files within the database itself using PL/SQL packages, and then 
using a generic file extraction program to handle the details of putting ASCII characters into Unix 
(or other) file systems. This has allowed us to reduce development time of programs to generate 
new file types, and greatly simplified supporting multiple platforms. 


Introduction 


At Rensselaer, we maintain a lot of our system 
and site configuration information in a database, using 
a package we developed call Simon [5, 7]. We are not 
alone in these efforts. I have talked with system 
administrators at many other sites who are working on 
similar projects, including the University of Alberta, 
Simon Frasier University, SUNY Albany, and the Uni- 
versity of Connecticut. One of the early projects of 
this type was at MIT’s project Athena and their Moira 
[15] package. This is not limited to educational sites, 
I have also spoken with folks at Cisco Systems and 
Collective Technologies about similar projects. A 
quick glance at the last few LISA proceedings show a 
number of similar projects including Accountworks 
[3], NFS Configuration Management [4], Unix Host 
Administration [16], Aurora [12], Exu [14] (Ok, they 
talked about doing it) and others. 


In practice, a number of configuration files need 
to be extracted from the database. These files range 
from host specific files such as /etc/printcap [8] to 
platform specific files like /etc/group to site wide con- 
figuration files like the resource record files for the 
domain name system [6], to HTML and LDIF files for 
the University telephone directory [10]. Traditionally, 
each file is generated by a file specific C program, or 
in some cases, a SQL*PLUS script. Depending on the 
specific requirements of the file, the program might 
deal with version control (generate only when infor- 
mation changed), trigger post processing, and have to 
run under different operating systems and environ- 
ments. 


The general pattem when a new file type was 
needed was to grab one of these existing programs, 
change part of it and recompile it (possibly on several 
different versions of Unix). As the number of different 
versions of unix grew, and we started working with 
non unix platforms, maintaining all the different ver- 
sions of these programs and getting them distributed 
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to the correct machines got to be more and more of a 
hassle. In addition, these programs varied widely. 
They used different methods of connecting to the 
database, supported different concepts of version con- 
trol, or had additional control options, so you would at 
times find yourself having built on the wrong “‘base,”’ 
and have to do even more work. 


As our systems became more distributed and 
specialized, maintaining and developing these custom 
programs became more challenging, as some of the 
target systems did not share file systems with the 
development systems, tools were not available, etc. As 
a result, this made what should have been trivial 
changes in a file format (like adding a newline 
between records!) into an hour or more of work. For 
example, when we wanted to build DHCP configura- 
tion files, we wanted to build them on our DHCP 
servers. Unlike our existing DNS servers (Solaris) 
which mounted our AFS file system, the DHCP 
servers (OpenBSD) do not even have an AFS client 
available. Our existing practice of storing the file gen- 
eration programs or even the configuration files in 
AFS space was not going to work. 


Building Files in the Database 


Our new approach to generating files is to gener- 
ate the file in the database itself then simply write a 
short program to dump the “‘file’’ out of the database 
and into the file system of the target machine. This 
allows all of the file generation code to be stored and 
maintained in the central database, using a consistent 
set of tools. In addition, the program to extract the file 
can be generic, and once built for a particular operat- 
ing environment, could be used for any of the files we 
might generate for that system. 


Our solution to this (see Figure 1), consists of 
three layers. The first layer is a generic program (Gen- 
erate_File) that runs on the target system that can con- 
nect to the database and write a file on the target sys- 
tem. This calls on the second layer: a PL/SQL package 
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[13]' File_Gen which runs on the database server. This 
module provides the sole interface to the database for 
the file generation program, and handles all of the 
access control for each of the desired files we might 
generate. This in turn calls the bottom layer for the file 
specific packages, such as Gen_RR_File, Gen_DHCP_ 
Config and Gen_Ldif that handle the details of extracting 
the data and formatting it for a given type of file. 


This has proven to be a very powerful technique, 
allowing us to keep the data extraction and formatting 
of the file in the database, while doing the actual file 
generation and version control management on our 
target systems. We can also use the Generate_File pro- 
gram for any number of different files, which means 
we only need to compile it once for any given plat- 
form. Adding a new file type involves writing a 
PL/SQL routine to format the data and registering it 
with the file generation package. 


General Operation 


To generate a file, the Generate_File program is 
executed with the file target as a parameter. The pro- 
gram connects to the database and calls the 
File_Gen.Get_Attr? stored procedure, passing it the file 
target. File_Gen.Get_Attr does some access checking, 
possibly calls a target specific procedure, and returns a 
file name, and optionally, some version information. 
The Generate_File program then opens the file and 
calls File_Gen.Get_Data routine, which calls a target 
specific package, and passes the result back up and the 
line 18 written to the file until a Null is returned. The 
file is then closed, and moved into place. 


For many of the files we generate, (/etc/printcap, 
letc/passwd, /etc/group, etc.), we just want a single file 
produced. In other cases, we want to generate more 
than one file at a time. For example, when we are 


‘PL/SQL Packages are a collection of functions, proce- 
dures, cursors and variables. These can be both public and 
private. Access to the public procedures can be granted to 
oracle users or roles. Once accessed in a session, a package 
maintains state between calls. 

2Procedures in packages are referenced as package 
name.object name. This is the Get_Attr procedure in the 
File_Gen package. 






Gen RR File 


[Generaerie | 


Gen DHCP Contig 


generating the resource record files for Bind, we want 
to regenerate all files that have changed. A single 
invocation of the program may create a lot of files. To 
handle multiple files, the Generate_File program then 
calls File_Gen.Get_Attr to get a new file name and the 
cycle is repeated until File_Gen.Get_Attr indicates that 
there are no more files. 


Generate_ File 


In our current implementation, the host program 
is written in C, using the Oracle C pre compiler 
(PRO*C). However, we are not limited to using this 
interface. We are planning on writing one in JAVA, 
and one could be written in PERL or any other lan- 
guage that can access the database. What is more, 
there is no problem in mixing and matching, all inter- 
faces can generate all file types. 


In our implementation, the program does a few 
more housekeeping details for us. When we open a 
file for writing, we actually open the file filename.new. 
Everything is written there, and after it successfully 
closes the file, we move filename to filename.old and 
then move filename.new to filename. This gives us a 
little bit of protection from full file systems and net- 
work interruptions. It is possible for other applications 
to have the target file open. The script calling Gener- 
ate_File program needs to be written to take the appro- 
priate action such as signaling the other process. 


The program has no special privileges. It is up to 
the caller to ensure that it is run in the proper directory 
with the proper access to the file system. In appendix 
1, we have the script? used to generate our telephone 
directory web pages. It is run as root via cron once a 
day. After setting some variables the script attempts to 
authenticate to AFS. The hosts.klog is a script we use 
to authenticate root procesSes on a system to AFS. 
Next we invoke a simple “locking” script* to ensure 
that we are the only copy running, cd to the target 
directory and generate the files. The program is 
invoked with the ‘“-nvr’” (short for Needs Vos 


3The /usr/vice/etc/pagsh is an AFSism, used to help control 
AFS authentication. 

4It is crude and depends on a file existing or not. But it has 
not failed us in six years of operation. 
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Figure 1: Service model. 
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Release) option. If we regenerate a file, we write that 
file name to the file specified by -nvr. After we are 
done with all the file generation, we check for the file 
specified by “-nvr” and if it exists, request a vos 
release.° Instead of a vos release, other post processing 
could be trigger such as a “‘make yp,” or whatever is 
needed. 


Our version of the Generate File program can 
connect directly to the database (if it is on the database 
machine), or can connect via SQL*NET to a remote 
database. In this case it can read an Oracle id and 
password from the command line or from a file. This 
works well in our environment. 

File Version Control 

One of the bits of information the program gets 
from the File_Gen.Get_Attr call is file version informa- 
tion. For example, we have around 170 different sub 
domains at our site. When we regenerate the resource 
record files, we don’t want to build the entire set each 
time, but rather, we want to just update the files that 
have changes. 


SRR take for rpiwedu 


Simon Database Version:26473884 


Generation Date: 23-MAY-2000 11:07 


Figure 2: Version Number in file. 


To this end, we maintain a version number for 
each file. In many cases, this is written in the first 10 
lines of the file with a particular flag string (see Figure 
2) where we scan for the string “Simon Database Ver- 
sion’ in the header comments of the RR file. For files 
that do not support comment lines, we write a parallel 
file with the string .vers. prepended to the file name; ie 
the version file for a file called passwd would be 
.vers.passwd. 


SThe “‘vos release” command is an AFSism that is used to 
reclone read only, replicated disk volumes. 


The File_Gen.Get_Attr routine returns, along with 
the file name to use, an optional version number for 
that file and an optional flag string. If the flag string 1s 
present, it will scan for it in the target file. If there is 
no flag string, then it will look for the parallel version 
file. The program then compares the database version 
number with the file version, and if they don’t match, 
the file is regenerated. The Generate_File program has 
a -force option to make it ignore the version numbers 
and generate all files. 


Post Processing Control 


With the version control enabled, it is possible to 
run the program for a given target and not write any 
files. In this case, no post processing would be needed. 
To assist in this, another option lets you specify a flag 
file. If a file is written, its name will be written to the 
flag file. Post processing code can check to see if the 
flag file exists and take appropriate action. We use this 
to automatically replicate files in AFS after genera- 
tion. This is shown in appendix 1. Another example 
might be generating the /etc/inetd.conf file. If you 
updated the file, you would want to restart inetd. If 
there was no change to the file, there would be no 
need to signal the inetd process. This feature was used 
in some of our older programs and proved useful so 
we kept it as we moved to this new approach. 


File Gen Package 


The core of this entire project is the File_Gen 
package, which acts as the gatekeeper. It provides 
access control and handling many of the interface 
details to the actual file generation procedures. 


Get_Attr 


When the Get_Attr procedure is called, you sup- 
ply the name of the desired target (and an optional 
parameter; see Figure 3). It first checks to see if the 
current Oracle user is allowed to generate that file 
(access the data). If they are, it returns a file name, 
some control flags and some version control informa- 
tion. If there is some problem (unknown target, no 





Name Type 
Target Varchar2 In 
ErrMsg Varchar2 Out 


| Filename Varchar2 Out 
be generated. 

DBMSOut Varchar2 Out 

Direction Varchar2 Out 


Version_Number Number Out 


Description 


Name of file set to be generated. 
If not null, display the message and terminate. 
Name of the file to be generated. When null, there are no more files to 


If ““Y’’, then DBMS _ Output routines may be used. 

Direction of transfer, ““GET” a file from the database, or ‘““PUT”’ a file 
into the database. 

The current version number of the file to be generated. If null, don’t 


check version before generating file. 


| Version_Str Varchar2 Out 
| ber. 


Par3 Varchar2 In 


If not null, scan the start of the file for this string and the version num- 


A parameter passed from the command line of the caller to the file gen- 


eration package. Usage defined by the individual package. 








Figure 3: Get_Attr parameters. 
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access), the error message is retumed instead. Get_Attr 
is the key routine which drives the entire generation 
process. Once a file is generated (or skipped due to 
version numbers matching), this routine should be 
called again for the next file. 


In our initial implementation of the File Gen 
package, we used a case statement® This required 
changing and recompiling the package every time a 
new file target was added. This also made File_Gen 
dependent on all of the file-specific packages. Fortu- 
nately, Oracle provides a dynamic SQL package called 
DBMS SQL [1] that allows a PL/SQL procedure to 
parse and execute an arbitrary PL/SQL routine. With 
these details worked out, adding a new file target was 
reduced to making an entry in the Generate_File_Types 
table (and writing the file-specific package of course); 
see Figure 4. 


While the general file generation model has the 
File Gen.Get_Attr called until there are no more files to 
be generated, many of our file targets only produce a 
single or fixed set of files. Having to write a 
Get_Attr_Rtn that keeps track of the current state (Start- 
File or EndFile), retums the file name, the version 
string, and so on was a bit of a hassle. So, if the 
Generate_File_Types.Get_Attr_Rtn is null, the File_Gen. 
Get_Attr will ‘“‘fake it,” using the values supplied in the 


6Actually, PL/SQL does not have a case statement, so it 
was a large if/elsif/elsif statement. 


Finke 


table. This makes it very easy to add simple file set 
targets. By using different sequence numbers, multiple 
files can be generated without having to write a file 
specific Get_Attr_Rtn. We use this to generate both our 
people and department LDIF files for our LDAP 
server. 


Set_Program_Version 


Set_Program_Version is used to make host infor- 
mation (current user, hostname, program version) 
available to the generation routines. We frequently 
include version information and other diagnostic info 
in the header of a file (see Figure 5). This 1s called at 
the start of the run, before any calls to Get_Attr or other 
routines. 


Get_Data 


Once a file is opened, the Get_Data routine is 
called and the result is written to the output file. If the 
DBMS OUTPUT flag was set by Get_Attr, then the 
Dbms Output buffers should also be written to the 
output file. This cycle repeats until Get_Data returns 
null. DBMS_ Output [2] is a package supplied by Oracle 
that will buffer text provided in calls to DBMS_Out- 
put Put_Line until the application retrieves the text via 
calls to DBMS_Output.Get_Line. 


Put_Data 


When we are “PUT’’ting a file into the database, 
we Call this routine for each line of the file. When we 


Description 


32 


The name of the target file set. 

Used to order files within a file set. 

Oracle role required to process this file. 

Name of the file set specific get attribute routine. 

Name of the stored procedure to be called to retum a line of 


Name of the stored procedure to be called to put a line of the 
file into the database. 

Name of the stored procedure to be called when all lines of the 
file have been read. 

Default file name to be used 1f Get_Attr_Rtn is null. 

When “Y’”’, indicates that this file target may generate output 
via the DBMS_OUTPUT package. 

Default direction to be used if Get_Attr_ Rtn is null. 

File version string to be used if Get_Attr_Rtn is null. 

File version number to be used if Get_Attr_Rtn is null. 

A version number (if any) for the package being used to gen- 
erate the file. We manage our stored procedures with RCS and 
use the “$Id$” value here. | 
A file reference to the source file used to create this package. 
We use the ‘“‘SHeader$”’ value here. 

The date when this file target was first made available. 
The date when this file target was most recently changed. 


Name Type Size 
Target Varchar2 32 
Seq_Number Number 
RoleName Varchar2 255 
Get_Attr Rtn Varchar2 65 
Get_Data_ Rtn Varchar2 65 

the file. 

Put_Data_Rtn Varchar2 65 
End Data Rtn Varchar2 65 
Def_Filename Varchar2 65 
DBMSOUT Varchar2 l 
Direction Varchar2 4. 
File Version_Str Varchar2 64 
File Version.Number Varchar2 32 
Package_ Version Varchar2 128 
Package Header Varchar2 255 
Create Date Date 
Update_Date Date 
Comments Varchar2 255 


A short description of this file target. 


Figure 4: Generate_File_Types table. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Finke An Improved Approach for Generating Configuration Files from a Database 


have reached the end of the file, we call the End_Data 
routine which signals the database to do any post pro- 
cessing required on the data that was just loaded. Both 
Put_Data and End_Data have an ErrMsg parameter. If 
this 1s not null, the message should be displayed and 
the file load terminated. This allows the underlying 
file load packages to signal fatal exceptions. 


Generate_File_Internal 


There is a second PL/SQL package called Gener- 
ate_File_Internal. From a strictly programming stand- 
point, this should be part of the Generate_File package. 
However, the Generate_File package is permitted to the 
public, thus all public procedures are available for 
anyone to use. Since the Generate_File_Internal defines 
some procedures to define and change file targets, it 
would not be good to let just anyone access these. 
Rather than put specific access control inside these 
routines, we kept them here and use Oracle access 
control to limit who can define new targets. 
Add_Target_Complex 

This entry point is used to add a complex file tar- 
get, that is, one that provides a Get_Attr_Rtn to supply 
the file name, version info, etc. By using a procedure 
to add entries to the Generate _File_Types table, we are 
able to do a bit of sanity checking and maintain the 


4 Simon Database Version: 26512881 


create date field. If there is an existing target with the 
same sequence number, it 1s replaced with this one. 
You will note that the ordering of the parameters (see 
Figure 6 for a list) is different from the earlier proce- 
dures and table definitions. Some of the parameters 
are optional, so these are left to the end of the parame- 
ter list. 


Add_Target_Simple 


This routine is used to define simple targets. 
Ironically, this particular call is more complex than the 
previous call. Of course, since we are off-loading 
more of the work on the Generate_File package, the 
trade-off is ok. Like the previous routine, some of the 
parameters (see Figure 7) have been moved around a 
bit, and some of the later ones are optional. 


There is also a Add_Target_Put which is similar to 
the Add_larget_Simple, except it defines a simple 
““put’’ entry rather than a simple “‘get”’ entry. 


Target=LIST 


In order to test these routines, as well as add 
some handy tools, this package also defines a “LIST” 
target that lists all the names and descriptions of all 
define targets. This can be handy when you need to 
generate a file and can’t remember the target name. 
(This was a very annoying lack in the previous 


li Generation Date: 30-MAY-2000 18:12 
+ Generation Hostname: vemr-42.server.rpi.edu 
+ Program Name:../Generate_File 
# User Name: finkej 
dE Generate_File: $Id: Generate_File.pc,v 1.3 
Ft 2000/05/13: 23:94:20 finke} Exp finke,:s 
Hf Generate_DHCP_Files: $Id: Create_Package_ Generate_DHCP_Files.sql,v 
H at 20007 C5711. 3232 10ils tinkeqs 
+ Host_Dns_Maint: $Id: Create_Package Host_DNS_Maint.sql.v 
# 122: 2000/05/11 -22°50e17 finkey Exp tinkey 1S 
+ Host_Maint: $Id: Create_Package Host_Maint.sql.v 
Ht 1.2 2000/04/28 -20707*00 finke;] Exp finke}. S$ 


Figure 5: Version information. 


Description 


Name Type 
Target Varchar2 ‘The target file set name. 
| RoleName Varchar2 


| given target. 


The role required. Note, this is checked only for the first entry for a 


The version number of the package (“‘$Id$’’). 
The path name of the source file of the package. 


The routine name. Since this is a complex target, this must be provided. 
The get data routine name. Technically, this could be an optional pa- 


rameter, but so far all complex targets have one. 


An optional sequence number. If not specified, it will default to 10. 


(Optional) An end data routine name. | 


Pack_Version Varchar2 

Pack_Path Varchar2 

Comments Varchar2 Comments on the package. 
Get_Attr_Rtn Varchar2 

Get_Data_Rtn = Varchar2 

Seq Number Number 

Put_Data_Rtn  Varchar2 (Optional) A put data routine name. 
End Data Rtn — Varchar2 


Figure 6: Add_Target_Complex parameters 
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version.) There is also a version of this that lists pack- 
ages and the location of the source for each of them. 


File Specific Packages 


Once the upper two layers are installed, the rest 
of the work takes place at the file specific package 
level. Up to this point, we did not really know nor care 
what files we were generating or reading. At present, 
we use this system to generate resource record files for 
BIND, host files, DHCP configuration files, 
/etc/passwd, /etc/group, white pages directories in 
ph,ldif and CSV formats and some web pages. We 
also use it to load in accounting records from our 
backup system and directory information from a 
remote campus. This list will be growing over time. 


A rather simple example of this, is the package 
to generate the /etc/passwd file. A source file to create 
this package is available in appendix 2. Our practice is 
to have a source file that we feed into SQL*PLUS, the 
Oracle SQL interface. To facilitate test versions, we 
use a “define name=”’ statement to set the package 
name. This is substituted in the appropriate places in 
the file when executed in SQL*PLUS. During develop- 
ment, we can change the name and not risk wiping out 
the production version. 


The source file has three sections: defining the 
package specification; the package body and finally, 
registering the new target with the Generate_File rou- 
tines. 


The prompt statement prints the message. This 
proves helpful when there are errors, as you get a bet- 
ter idea which section had the failure. The first sec- 
tion, defining the package specification (Create or 
Replace Package &name) is pretty simple. It includes an 
RCS ‘“$Header$” and the definition of the Get_Data 
routine. The parameters Pl and P2 are optional 
parameters that are passed in from the command line. 
Their usage is based on specific package. They might 
be used to enable debugging or in some way control 
details of the file generation. 





The second section, defining the package body, 
(Create or Replace Package Body &name) has the actual 
PL/SQL code that is executed to produce the password 
file. The cursor sets up the query to extract the data. 
This happens on the first call to Get_Data when the 
cursor is opened. Each fetch statement brings the next 
row of data into the “R” variable. This record is auto- 
matically defined based on the columns in the select 
statement. The quoted strings in the select statement 
are column aliases. Finally, when the end of the data is 
reached, a null is returned rather than a PW file entry 
and the cursor is closed. 


The third section is used to register this file type 
with the Generate_File routines. Since this is a simple 
case, we can use the Add_larget_Simple routine, define 
the target name, skip access checks, include the ver- 
sion info and some comments and set the output file 
name to be passwd_demo. Finally, we provide the 
name of the get data routine. We do not specify a 
sequence number, so this will take the default of 10, 
and replace any existing definition for  etc- 
passwd_demo. 


Conclusions and Futures 


Execution Time 


In some casual timing tests we found mixed 
results. For a simple extraction, where all data is in a 
single table, the custom program appears to be slightly 
faster than the PL/SQL approach. However, if the file 
is complex, requiring data from many tables, the 
PL/SQL approach is faster. This is in part because 
more of the work taking place on the database server, 
thus reducing the I/O between the server to the appli- 
cation. 


There is also some overhead from using the 
Dynamic SQL codes rather than hard coding the 
switch statement. Some sample runs show about a 
15% increase in run time by using the dynamic sql 
routines. However, given the reduction in develop- 
ment time, this seems worth the tradeoff. If this 





Name Type 
Target Varchar2 
RoleName Var char2 


Pack_Version _—‘- Varchar2 
Pack Path Varchar2 
Comments Varchar2 
Filename Varchar2 
Get_Data_Rtn  Varchar2 
Seq Number Number 
to 10. 
DbmsOut Varchar2 
File_Vstr Varchar2 
File Vnum Varchar2 


Description 


The target file set name. 

The role required. Note, this is checked only for the first entry 
for a given target. 

The version number of the package (““$Id$”’). 

The path name of the source file of the package. 

Comments on the package. 

The name of the file we will generate. 

The get data routine name. 

An optional sequence number. If not specified, it will default 


The value for the DBMS Output flag. Should be ““Y” or “N’’. 
An optional version string to be used with version control. 
An optional version number. 


Figure 7: Add_Target_Simple 
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difference is a problem, it would be possible to write a 
program to generate the source file. 
Development and Deployment Time 


The development time to add a new file to be 
generated to the system has been significantly 
reduced, in part because all of the changes are now 
isolated to the central database, rather than being dis- 
tributed across many remote servers. Once the Gener- 
ate_File program is installed on a machine, virtually no 
work needs to be done on that machine. 


The PL/SQL language is well integrated in the 
Oracle environment, and many of my PL/SQL pro- 
grams are noticeably shorter than the PRO*C pro- 
grams they replaced. For example, the original 
PRO*C version of the program to generate the fac- 
ulty/staff phone directory was 2639 lines of C/PRO*C 
code versus the 1588 lines of PL/SQL that replaced it. 
I feel that the PL/SQL was faster to write as well as 
being shorter. 


Another factor that reduced development time, 1s 
that we often are doing a web component for the front 
end of these files. This results in a set of PL/SQL rou- 
tines to access the tables, so half the work is done by 
the time we need to write a file generation routine. 


Another advantage is that we are able to use dis- 
play code used by the Oracle web server when we are 
generating static HTML pages. We are using this con- 
vergence to dynamically preview static web pages 
which reduces the development time with those as 
well. 


Handy Tricks 


To help validate post processing, when we gener- 
ate our RR files for bind, we include an entry at the 
end that includes a ‘“‘wc’”’ for the entire file in one of 
the fields. This allows post processing scripts to 
quickly determine if they are working with a complete 
passwd file, and not one that got truncated. Adding 
support for this to the system could be handy. 


For our generated password files, we include a 

special entry at the end (show here folded): 
TIMESTMP : #HFTIMESTMP : 397: 4000: 

13834 37930 1236688 

2000-09-20-13-01-07:/tmp:/bin/true 
The GECOS field has a “‘wc”’ value, as well as the 
time and date of generation. This is handy when your 
password file distribution processes hit snags. 
New Platforms 


We are in the process of developing a JAVA ver- 
sion of this program to run on Linux systems. We 
don’t have the Oracle development tools licensed for 
this platform, and our network servers are moving 
away from our centrally administrated AIX machines 
to this platform. This is actually being driven by our 
security team wanting to process the DHCP lease file®. 


"Unix wordcount; returns lines, words and bytes in a file. 

8It is somewhat ironic that this paper started with describ- 
ing how we generate DHCP configuration files and has 
come back around to the other side of that project. 
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New Directions 


Since we already support “GET” and “PUT” 
operations, what else could we want? Well, rather than 
getting data from a file, how about getting data from a 
program. We currently have a custom C program that 
runs |scfg on a pipe, captures the output, digests it a bit 
and saves it into the database. We could replace it with 
a file set that does the same thing. What is more, if we 
want to add additional program runs, we can do that 
by changing the package stored in the central database 
and we no longer have to worry about getting new ver- 
sions of the custom programs out to our clients. I 
expect that we will have our client machines running 
Generate File with the file target of ‘“Self Exam”’ peri- 
odically to bring information on the config back to our 
central management system[11]. One limitation of this 
is that it only works with text files. Trying to pick up 
binary files such as /var/adm/wtmp might still require 
custom programs such as the one we wrote to assist in 
doing demographic analysis of workstation use [9]. 


Along with running programs and capturing their 
Output, we could run the other way and generate text 
and pass it to a program. This could be used as a quick 
and dirty interface to lpr or mail. One of the parame- 
ters available to the file generation packages is the 
platform (Hardware, OS) so it could presumably make 
reasonable assumptions as to what programs it can 
call, and where to find them. This support could also 
be used to initiate post processing, although that might 
better be handled in the calling script. 


Internal Changes 


I expect that we will merge the Gener- 
ate File_Internal package back in to the Generate_File 
package and build the access checking into the add 
target routines. 


References and Availability 


All source code for the Simon system is avail- 
able on the web. See http:/\www.rpi.edu/campus/ 
rpi/simon/README.simon for details. In addition, all 
of the Oracle table definitions as well as PL/SQL 
package source are available at http://Awww.rpi.edu/ 
campus/rpi/simon/misc/Tables/simon.Index html . 


While the specific file generation packages are 
Rensselaer specific, both the host programs and the 
File_Generate should be pretty generic and could be 
used at other sites with no modifications. 
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Appendix 1: DirectoryGen.sh 


#!/usr/vice/etc/pagsh 
# Script to update Directory files from Simon 


HTMLROOT=/afs/.rpi.edu/dept/acs/rpinfo/common/AutoGen 
DirPgmRoot=/campus/rpi/simon/directory/2.0/@sys/bin 
HTMLROOT=/afs/.rpi.edu/dept/acs/rpinfo/common/AutoGen 
LOCKPGM=SROOT/common/bin/wait_for_lockfile.sh 
LOCKEIL=SROOT/Dirgen-Syne= lock 
VOSRELEASE=SROOT/common/bin/SQLVosRelease.sh 
HNVR=SHTMLROOT/Needs_Vos_Release 


4A Define Oracle variables 

LOGNAME=SimonxXfr ; export LOGNAME 
ORACLE_HOME=/opt/app/oracle/product/8.0.5 ; export ORACLE_HOME 
ORACLE_SID=SIM3 ; export ORACLE_SID 
PATH=SORACLE_HOME/bin:SPATH ; export PATH 


4#F Get a token 

if | =e /ust/local/erc/host-kloc |) "then 
/usr/local/etc/host.klog -t 

else 
echo "Unable to get token" 
exit 1 

at 

7 Ger a Jloek 

ak C SSEOCKPGM SLOCKEIL. dO.) 

then 
echo “Unable to: Lock SLOCKFIL" 
exit. | 

real 

dE Regenerate directory HTML files 

cd SHTMLROOT 


dirdeptgen=$DirPgmRoot/Generate_File 
dirdeptpar="-target department_html -nvr SHNVR" 
if [ -x S$dirdeptgen ]; then 

su SLOGNAME "-c Sdirdeptgen Sdirdeptpar" 
else 

echo "Unable to run $dirdeptgen" 

exit l 
fe 
## See if we need a vos release 
it, Sf CHNVR «5 


then 
echo "==>[ignored: n]<== Vos releasing for" 
cat SHNVR 
af [| =x: “SVOSRELEASE | 
then 
SVOSRELEASE simongen 
rm SHNVR 
else 
echo "Unable to locate sysctl, vos release aborted" 
fd: 
Tea 


Appendix 2: Generate_Passwd Package 


define name=GENERATE_ETC_PASSWD_DEMO 


prompt Creating package &NAME 
create or replace package &name as 


=e: $Header: /afs/.rpi.edu/campus/rpi/simon/prop/2.0/init_sql/RCS/ 
Create_Package Generate_Etc_Passwd_Demo.sql,v 1.1 2000/09/20 
Ole lie41 finkej] Exp. finkes <s 


-- Generate a /etc/passwd file from the logins table 
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procedure eet dataCresult out, varchar? plan varchar2,-:p2,.invarehar2):; 


end &NAME; 

/ 

prompt Creating package body &NAME 
create or replace package body &name as 


Cursor Get_Pw_Ent_Curs is 
select Username. "UNAME™| Unixuad: “UTDY. mvi(naxeid: 4000) ° “Gip", 
Publsie Personal ~[nto.'GECOS” 
from Logins 
where source like ’PRIMARY%’ 
and when_marked_for_delete is null 
order by unixuid; 


procedure eet data (result, out -varchar2, pl an varcehar2..-p2 din varchar?2) 
is 
R Get_pw_Ent_curs%Rowtype; -- The data from Oracle 
begin 
-- First time through, we open the cursor 
if not Get_Pw_Ent_Curs%IsOpen 
then 
Open Get_Pw_Ent_Curs; 
end if; 
-- Get the data into R, a record based on the query 
Fetch Get_Pw_Ent_Curs 
Into: KR; 


if Get_Pw_Ent_Curs%NotFound 


then 
“=~ Had On Gata, «Chose. -— he icursor rana rerun ta pill 
Restlt- osu 
Close Get_Pw_Ent_Curs; 
else 
-- Build the password file entry 
Result *R.Uname ||| ’¢*z’ | | to_char(R.Uid) || '3’ 
ReGidvilii ce ill RvGecossi| (ei? 
“homey ln ltram couchar (Reid mods 100;, 0.9). Vi “a lil SRecuname 
i) “s/bin/ bash’: 
phd af: 


end get_data; 
end &name; 


/ 
-- Call Add_Target_Simple to register this file target. 
begin 
Generate Pile Internal Add-Targer Simple 
"etcpasswd_demo’, -- Target Name 
real, -- Let ANYONE do this 
‘$Id: Create_Package Generate_Etc_Passwd_Demo.sql,v 1.1 2000/09/20 
Olli Al tanke? Exp. fankey iS, 
'SSouirce: /ats/.rpi.eduy/ campus/rpi/ simon/prop/ 2.0 /init-sql/ROG/ 
Create_Package Generate_Etc_Passwd_Demo.sql,v $’, 
‘Generate a> f/ertc/ passwd files -a demo for isk’; 
*passwd_demo’, -- Output File Name 
’&name..Get_Data’); -- Routine Name defined above 
end; 
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Fokstraut and Samba — Dealing with 
Authentication and Performance Issues 
On A Large Scale Samba Service 


Robert Beck & Steve Holstead — University of Alberta 


ABSTRACT 


At the University of Alberta, we have approximately 55,000 user id’s using central services 
authenticated by Kerberos. We use AFS for central file service. We use Samba to provide 
Windows compatible access to much of our central file service. Samba contains a number of 
useful features for Microsoft Windows compatibility, including a kludge to deal with the problem 
of Windows sending an all uppercase version of a user’s password. We observed that when 
Windows connects to a share, it frequently attempts many incorrect passwords repeatedly before 
trying the correct one. This created a very heavy authentication load on our central Samba service 
when users would connect every morning and authenticate. We observed this load and noticed that 
most of our problems were caused by repeated attempts to authenticate, and the high cost of 
checking these attempts. 


To help reduce the load due to authentication, we implemented FOKSTRAUT, a set of 
modifications to Samba to cache recent password failures and successes in a DBM database built 
by the Samba server as it runs. By caching the recent failures we avoid expensive re-checks of the 
(many) other passwords Windows likes to send us. We also cache the correct case of the real 
password, and by doing so we avoid the expensive overhead of “cracking” an all uppercase 
password When Windows decides to send one. We also use FOKSTRAUT to cache the NT and 
LanMan password hashes of a users password once we see a successful authentication. This then 
allows us to use the newer Windows NT password hash after the user has connected once, without 
having to centrally convert and maintain a large SMB password file, and while maintaining the 
ability of our server to access services such as AFS which can not be authenticated against using 
the Windows password hash alone. Performance on our service has been drastically improved 


since the implementation of FOKSTRAUT. 


Introduction 


Our experience started with a performance prob- 
lem. We are a large site which uses Kerberos [1], 2] 
with approximately 55,000 user ID’s. We have a large 
scale Samba [4] service which serves up files from our 
central AFS [3] file service using the Microsoft Win- 
dows (Windows) [9] SMB protocols [5, 6]. It was and 
is exceedingly popular with our on-campus clients 
who use it to access centrally maintained file space 
when they must use a Windows machine. The problem 
we had was that while the service would perform well 
during the day, it would fall over under a crippling 
load in the mornings when users would establish their 
initial connections. This problem hit us rather sud- 
denly this last year when various factors resulted in a 
large increase in use at the start of a school term. We 
then set out to determine the source of the problem 
and what, if anything, we could do about it. 


The Source of our Problem 


We examined the server at various times, and we 
found that it was completely CPU bound in the morm- 
ings when users were connecting, with many individ- 
ual smbd processes competing for CPU. Once this had 
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been determined, we set out to find out where it was 
spending its time by adding a few key little diagnostic 
printfs to our Samba code. Watching the users con- 
nect in the morning we found the source of our CPU 
hogs. 


On connection, many of our users’ machines 
would try several times with a password that obvi- 
ously wasn’t their password on our service. Several 
attempts had to be made, and fail, before the correct 
password would be given by the Windows client 
machine. The result of this was that our server spent a 
lot of time failing the same password, for the same 
user, Over and over again. 


Our Kerberos passwords may only be changed 
through a password changing mechanism that forces 
the use of “‘good”’ passwords, and does dictionary and 
pattern checks against any attempts. Very frequently 
On Our server, users would attempt to connect initially 
with very bad, easy-to-guess passwords, and then only 
after several failures, would try the correct password. 
These initial passwords which the connection would 
attempt and fail were the same as the users “Win- 
dows” password (NT domain or otherwise) with 
alarming regularity. So, by configuration or design, 
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many users’ Windows machines were determined to 
tell us their local passwords before attempting to give 
us the one we wanted to authenticate them with Ker- 
beros. This has interesting security implications — it’s 
easy for our clients to configure Windows to leak their 
passwords to an SMB server. For the purposes of this 
exercise, we were only concerned about the perfor- 
mance implications. 


In addition to the repeated attempts of an incor- 
rect password, many of our users’ Windows machines 
will attempt to use an all uppercase password. We 
have little or no control over the client software ver- 
sion, and so were not able to avoid this problem at the 
client end. This means we have to support the pass- 
word level [10] ‘‘crack”’ feature in Samba where the 
server would repeatedly try different case combina- 
tions. While we knew this was a possible cause of the 
heavy load generated by authentication, the effect was 
greatly magnified by the repeated attempts of incorrect 
passwords — each incorrect password had to be 
checked repeatedly in multiple cases. 


We were faced with a huge overhead on authenti- 
cation which involved ‘“‘cracking’’ passwords, and 
doing it repeatedly on the same bad password. Short 
of abandoning our Windows clients we had to find a 
way to deal with this on our Samba service. 


The NT Password Hash 


Samba itself supports the Windows NT 
encrypted password scheme. To use this, a UNIX 
administrator creates an “smbpasswd’’ file [11] in 
which the Windows NT hash of the user’s passwords 
are stored. In this method, the client machine passes 
the Windows NT style hash, and is not faced with the 
problem of case insensitive password. One possibility 
we examined was that of converting our service to 
require this scheme. We could not do this for two rea- 
sons: 

1. We maintain our central ID’s in Kerberos, We 
did not want the added administrative cost of 
maintaining a 55,000 user flat file of SMB 
passwords. 

2. We access AFS file space from this server — 
There would be no way for the server to get an 
AFS token to access the user’s file with only 
the NT password hash (and not the user’s Ker- 
beros password or a ticket). This will be a 
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problem with any external service that can not 
be authenticated against using the NT password 
hash. 


Our Solution: FOKSTRAUT 


Having ruled out doing it the “correct” way for 
Windows by using the NT password hash, we tried to 
examine what would give us a solution for our envi- 
ronment. First, we made a few observations: 

1. The users that handed us passwords that failed 
would usually try them twice before proceeding 
to the next password (sometimes another bogus 
one, and sometimes the right one). This was 
what most of our CPU bound processes were 
doing — busily cracking a bogus password. 

2. If we knew the real password, and were given a 
CaSe-insensitive version of it, we could start by 
checking the “correct” case and avoid the 
expense of repeatedly trying to crack the pass- 
word. 

3. If we knew the real password we could also 
compute the NT password hash. 

Knowing this, we implemented FOKSTRAUT. FOK- 
STRAUT is a password success and failure cache for 
Samba. It uses a DBM database indexed by the user 
name to store the most recent passwords that failed, as 
well as the last password that worked for each user. It 
stores the successful passwords both as the clear text 
password, and as it’s Windows NT hash, enabling the 
FOKSTRAUT cache to be used to authenticate Win- 
dows clients passing a Windows NT password hash 
for a user that has been seen before. Since it stores the 
clear text password along with the NT password hash, 
the server then knows the clear text password of a user 
authenticated with the NT password hash and can use 
this to authenticate the user for other services (AFS in 
our case). 


How FOKSTRAUT Works 


When our Samba server starts up, it checks for 
the FOKSTRAUT database, a DBM database indexed 
by user name. This database is created if it does not 
exist. The Samba server then stores and retrieves from 
this database each time a user authenticates. 


FOKSTRAUT keeps track of the following 
information: 
e The three most recent failed passwords, and a 
count of how many times each password has 


Sep 28 07:23:41 samba smbd[76722]: trying password AUCTIONS for user luser 

Sep 28 07:24:08 samba smbd[76722]: Bogus password, user luser, password AUCTIONS 
Sep 28 07:24:11 samba smbd[76722]: trying password AUCTIONS for user luser 

Sep 28 07:24:47 samba smbd[76722]: Bogus password, user luser, password AUCTIONS 
Sep 28° 07:24:49 samba smbd|(76722)% trying. password AUCTIONS for ‘user Tuser 

Sep 28 07:25:18 samba smbd[76722]: Bogus password, user luser, password AUCTIONS 
Sep 28 07:25:20 samba smbd[76722]: trying password Ac94metoo for user luser 


Sep 28 07:25:20 samba smbd|76/722)]: worked first time for user luser, 


Figure 1: Example of our diagnostic log output. Note the times between success and failure: during this time this 
smbd was busy using CPU attempting combinations of this password. 
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password Ac94metoo 
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been given and failed since the last successful 
connection. 

e The last successful password, stored as clear 
text, NT Hash, and LanMan hash formats. 

e An “smbpasswd” entry [11] for the user — this 
is used so that we can take advantage of 
Samba’s NT and LanMan password features if 
we find a user in the database. 

When a user connects to our FOKSTRAUT modified 
Samba server, the server will look up and retrieve the 
information about previous connection attempts in the 
FOKSTRAUT database. It then compares the pass- 
word the user is attempting to any previously cached 
failed passwords. If that password is the same as a pre- 
viously cached failure, the connection is immediately 
failed without checking against the system, and the 
failure count for that password is incremented in the 
database, and saved for the next time. When the fail- 
ure count for a failed password reaches three, this 
password is removed from the database. This ensures 
that a cached password failure can not continuously 
fail without being re-checked against the system 
authentication methods. 


Assuming the password didn’t match one of the 
cached failures, the password is then case insensitively 
compared to the password recorded as being success- 
ful previously. If the passwords match, the correct 
case stored in the database is used, and authentication 
proceeds against the system authentication methods. If 
authentication is then successful, this connection suc- 
ceeds. All previous failure counts are cleared back to 
zero, and the resulting record is stored back into the 
database for the next time. 


If the password presented doesn’t match any- 
thing FOKSTRAUT knows about we attempt to 
authenticate in the usual manner. If the password suc- 
ceeds, the correct case is recorded in the cache, the NT 
and LanMan hashes are computed, and the entry is 
saved as in the above case. If the new password fails, 
it will be added to the cache of failed passwords, 
replacing the least used (least failed against) of the 
three saved entries. 


If our modified Samba service receives an NT or 
LanMan hashed password, it checks it against the 
entry saved in the FOKSTRAUT database. If it is pre- 
sent, an smbpasswd entry from the database is used 
for the user, and authentication proceeds as it would in 
the usual case of an smbpasswd based connection in 
Samba [4]. If that authentication then succeeds, in our 
case, the server then gets Kerberos Tickets and an 
AFS token for the user based on the saved clear text 
password from FOKSTRAUT. At this point if both the 
NT hash authentication and the Kerberos/AFS authen- 
tication are successful, the connection is deemed to be 
successful. Otherwise, it is considered a failure. and 
the cache entries (including the password hashes) are 
cleared. 


The choice we made of saving three different 
cached failures was based on our own observations of 
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what we saw from our users. Three different bogus 
passwords failing was the most we typically saw 
before seeing the real password. The choice of failing 
against a cached failure up to three times before we 
checked again against he system was based both on 
our observations of the typical number of tries we 
would see, along with a desire to make sure we didn’t 
hold up a user unduly in the (unlikely) case where 
they make a connection attempt, the password fails, 
and they then change their password to what they just 
recently failed with. 


Advantages 


We have seen two primary results: 

1. We have seen a huge performance improve- 
ment, we can now easily deal with the several 
hundred users we get every morning where we 
could not before. Simple load average during 
peak times has dropped from peaking at 30 to 
peaking at 3, with a sustained average dropping 
from over 10 to less than 2. 

2. ) We now have a mechanism by which we can 
support the non-cleartext password SMB con- 
nections to our server, while still making use of 
Samba as a gateway to AFS, Users are simply 
told to connect once from a login server, or 
with a Windows machine set to send cleartext 
passwords on connection, and then after that 
the non-cleartext SMB connection will work. 

We are very happy with the increased level of perfor- 
mance these modifications have given us. 


Date rung SZ *s*runoce 
Tue Feb 29 08:30:05 2000 3.700 80.000 
Tue Feb 29 08:40:05 2000 8.900: TOOL 000 
Tue Feb 29 08:50:05 2000 10.800 100.000 
Tue Feb 29 09:00:06 2000 12.600 98.000 
Tue Feb 29 09:10:06 2000 17.500 100.000 
Tue Feb 29 09:20:14 2000 21.400 972000 
Tue: Feb: 29 09302512000 4.500 o72.000 


Figure 2: Run queue size and utilization, 8:30 AM to 
9:30 AM, week before implementation. 


Date rung-sz ‘%runocc 
Tue Mar 7 08:30206 2000 L700 37000 
Tue Mar 7 08:40:06 2000 1.400 48.000 
Tue Mar 7 08:50806 2000 ee = O18) 55:.000 
Tue Mar 7 09:00:06 2000 1.800 60.000 
Tue Mar 7 09:10206 2000 1.400 37 000 
Tue Mar 7 09:20:06 2000 2.000 d3°).000 
Tue Mar 7 09:30306 2000 L800 28.000 


Figure 3: Run queue size and utilization, 8:30 AM to 
9:30 AM, week after implementation. 


Disadvantages 


FOKSTRAUT has several disadvantages. The 


first one that comes to first one that comes to mind 1s 
security. If a Samba server is compromised and is run- 
ning FOKSTRAUT, all the users’ passwords are avail- 
able to the attacker through the database. This is a 
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serious concern to us, and we therefore ensure that 
FOKSTRAUT is only run on a dedicated, secured 
machine running no other services and that there is no 
access by regular users to the machine. We judge this 
risk as acceptable to us, given that even if the machine 
were not running FOKSTRAUT, an attacker compro- 
mising the Samba server could install a trojaned dae- 
mon and collect the same passwords with very little 
extra effort. The results of a compromise of the Samba 
server would be bad for us with or without FOK- 
STRAUT, considering that we must run Samba with 
cleartext passwords enabled. 


The other drawback from the point of view of the 
Windows NT passwords is that the server does not 
know the Windows NT hash until the client has suc- 
cessfully authenticated to it once so that the server 
knows the cleartext password. We get around this by 
providing a simple script on a login server in which a 
user can make an SMB connect and authenticate 
themselves, making themselves “known” to the 
Samba server. Alternatively, the users can make an 
initial connection to the service from a Windows 
machine set to send the clear text password, then 
switch to using the NT password hash. 


Conclusions 


A password cache is a major performance win 
when used with Samba service that must support clear 
text passwords and can not rely exclusively on the NT 
password hash protocol. In our case, with the ability to 
use the NT password hash for authentication and still 
have the server know the real password, it enables us 
to use the NT password hash connection mechanism 
while still supporting our external authentication 
mechanism (AFS) which is incompatible with the NT 
password hash. The performance wins we saw 
changed our service from one that would collapse 
under the load to one that now remains up for months 
at a time, providing reliable file service. If you are in a 
position where you have to offer large scale Samba 
service, we think this is definitely something to con- 
sider. 


Availability 


Our code is available as a patch for a current 
Samba distribution. It has been tested and used on 
OpenBSD [7] and AIX [8] (and should port very eas- 
ily to anything else remotely Unix-like). It 1s made 
available under BSD style license terms, and can be 
obtained at ftp://sunsite.ualberta.ca/pub/Local/People/ 
beck/fokstrau t/. 
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Designing a Data Center 
Instrumentation System 


Bob Drzyzgula — Federal Reserve Board 


ABSTRACT 


This paper describes the author’s efforts in designing an external, out-of-band hardware 
monitoring and control system for use with microcomputer-based server, storage and 
communications systems deployed in a data center environment. This system, when complete, will 
consist of a collection of microcontroller-based monitoring nodes, one per monitored device. Each 
of these intelligent monitoring nodes will be able to keep track of several temperatures, power 
supply voltages, fan speeds, and various indicators of system activity. In addition, they will have 
the ability to control a monitored system under the direction of an administrator sitting at a web 
browser. As of this writing, much of the initial research and architectural planning is complete. 
One prototype board has been built and shown to function as expected, and most of the required 
development tools and licenses have been procured. The hardware design for the first 
pilot/production board is largely complete. It is expected that these first boards will be built and 
assembled by late 2000, and so ftware development for this project will extend into 2001. 


Context and Motivation 

Origins 

The Automation and Research Computing Sec- 
tion (ARC), part of the Division of Research and 
Statistics (R&S) at the Federal Reserve Board (FRB), 
runs a reasonably extensive computer network for the 
staffs of R&S and of the Division of Monetary Affairs 
(MA). There are a total of about three hundred fifty 
persons in the user community. This “R&SMA 
Research Computing Network” consists of more than 
five hundred computers and over one hundred other 
network addressable devices — switches, routers, print- 
GIS, CIC: 


This network was installed in 1987, and con- 
sisted then primarily of Sun 3 servers which were scat- 
tered throughout the office areas in the supported divi- 
sions. Most users had VT220-type terminals on their 
desks, while a few researchers had their own Sun 
workstations. Over time, most desktop devices were 
upgraded to X terminals, but the servers — which were 
regularly upgraded and steadily increased in number — 
remained located in the end-users’ office space. 


Change 


In 1997, the Board’s central IT division replaced 
an aging mainframe and disk farm with newer models 
which took up a miniscule amount of space compared 
to the previous system. The then-barren raised-floor 
space was made available to “client divisions” as a 
place to put all the servers that were scattered hither 
and yon. R&S obtained space and resources sufficient 
to support thirty-five nineteen inch machine racks. 


In response to this and other challenges, a com- 
plete re-design and and re-implementation of the 
research network was begun, a project which included 
the migration of all servers to the newly acquired data 
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center space. This plan was carried out over a two- 
year period and was completed in mid-1999. 


Repercussions 


With this change, however, some of the remote 
diagnostic and control capability inherent in the previ- 
ous design had been lost. Many times the telephone 
had been a very effective system monitoring tool — 
users, strategically placed near servers, were often 
very effective in providing notification of problems 
such as noisily failing fans and disk drives, even with- 
out much in the way of training. Also, if a system was 
so sick that it would not respond to the network, these 
users were often able to reset that system on request. 
This was especially useful on weekends when support 
staff were answering calls from home. The data center, 
however, being a secure “glass house” environment, 
was off limits to the users and thus this was no longer 
an option. 


The new environment also resulted in some new 
types of problems. For example, we learned that, 
while one can equip a storage system with redundant 
power supplies, this is of limited use if there are no 
automatic mechanisms in place to provide notification 
of a failure of one of the supplies. We also leamed that 
the white noise in a data center, together with glass 
rack doors, can do a remarkably good job of masking 
even the piercing sound of a piezo-electric alarm two 
rows away. Finally, the ability to monitor AC power 
was lost when a new building engineer declared stan- 
dalone UPSs a fire hazard — the Fire Department 
needed a way to shut down the entire data center with 
a single switch, and the distributed UPSs made this 
problematic. A large, data center-wide UPS was 
installed in the sub-basement, but the ability to moni- 
tor individual circuit loads and power conditions was 
lost. 
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It became apparent that something needed to be 
done in order to gain better status information. 


Looking for solutions 


To recover our lost capabilities, we determined 
that a network-accessible instrumentation (or hard- 
ware monitoring and control) system was needed. 
While it is tempting to try to build a single system that 
addresses every problem, there are many advantages 
to breaking the problem up into a number of smaller 
bits that could be addressed individually. 


The No-brainer Stuff 


On this basis, we saw three areas which would 
have high immediate impact, were reasonably inex- 
pensive and easy or at least straightforward to do, and 
which addressed areas which resulted in severe sup- 
port difficulties. These three areas were serial console 
access, power control and graphical console access. 


Serial Console Access 


Several of the devices installed in these racks 
had serial consoles to which it was fairly straightfor- 
ward to provide remote access; the large number of 
serial ports, however, posed difficulties. 


Sun SPARC servers 


All of the production Sun servers, about 60 in 
total, are built from Sun Microelectronics mother- 
boards — either the quad-processor UltraA Xmp or the 
uniprocessor UltraAXi. One particular issue with 
regard to Sun SPARC machines and their serial con- 
sole is that, by default, these systems will halt when an 
RS-232 break signal is detected. 


As most people who work with Suns know, when 
one connects a serial console cable to the Sun, the 
resulting noise on the port often will generate a break 
condition. While the Sun’s halt-on-break behavior can 
be disabled in the Open Boot PROM, this comes at the 
cost of not being able to halt a system when required. 
By contrast, our goal was to be able to connect and 
disconnect terminals at will and still maintain the abil- 
ity to halt the system from the console. To solve this 
problem, a special “‘non-aborting”’ console cable from 
NuData — now part of MicroWarehouse [44] — was 
installed on each Sun server.! 


CMD RAID Controllers 


All of the RAID subsystems are again integrated 
in-house. We have approximately fifty RAID boxes in 
production, split between the SPARC and NT servers. 
The controller used in these subsystems is the CMD 
[14] CRD-5440. All configuration of this controller is 


'This device, — NuData part number NUD4273 — is a short 
console cable, one end of which contains circuitry to ensure 
that a break signal is never generated at the Sun’s console 
port except when one Is clearly being generated at the termi- 
nal end of the cable. Inspection of the circuit in this device 
reveals that it is actually fairly complex. It contains fifteen 
resistors, eight diodes, three transistors, one op amp and five 
capacitors — one of them a large 1000uF electrolytic. 
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done via a serial console connection on the back of the 
unit; the host knows not that it has a RAID controller 
attached. 


Serial port cabling infrastructure 


The first implementation of a system to connect 
to these serial ports, over one hundred of them, was 
based purely on 25-pair, Cat 3 Telco trunk cables and 
USOC patch panels. A patch panel was installed in 
every rack or two, and conventional, straight-through 
10-base-T patch cables were used to connect from 
those to custom-made serial null modem adapters 
[23]. The other ends of the trunk cables were brought 
into USOC patch panels. A few standard VT220-type 
terminals were installed on rack shelves; these could 
then be plugged into any port in those panels, switch- 
board operator style. 


Serial port server 


The switchboard system worked quite well, but 
the ultimate goal was to be able to connect to any of 
these ports remotely. To accomplish this, an X86 
Linux server was built and outfitted with Cyclades 
[18] multi-port serial cards. Shell scripts driving C- 
Kermit [16] were written to provide command-line- 
based access to the piles of serial ports provided by 
the Cyclades cards. In this way an administrator on the 
Linux serial port server could simply connect to the 
correct console without knowing which serial device it 
was on. This system could be used either at the Linux 
system’s console or through a secure network connec- 
tion from anywhere in the world. The multiuser capa- 
bility of the Linux box made it possible for several 
administrators to be working on separate systems at 
the same time. 


This of course represents something of a single 
point of failure in the support infrastructure for these 
machines; the backup system remains the old switch- 
board technique, since the Linux/Cyclades system 
uses the same patch panels. 


Power Control 


Each of the thirty-five machine racks was outfit- 
ted with a dedicated, 20A 120VAC circuit. To help 
manage these, Model 3302F intelligent power con- 
trollers from Pulizzi [56] were installed in each rack. 
These devices each have eight programmable power 
outlets, which can individually be turned off or on and 
can be programmed to turn on in a specific sequence 
with programmable time delays. These controllers can 
be daisy-chained on an RS-485 serial network, which 
then can be connected to a standard ASCII terminal or 
terminal emulator. 


This serial network was connected to the Linux- 
based serial port server described in the previous sec- 
tion, and scripts were written to interface to that serial 
network and generate the correct commands to, for 
example, power cycle a specific port. 


Graphical console access 


Windows NT, by default, does not support man- 
agement through a serial console in the same way as 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Drzyzgula 


do all these other devices. At present, we have about 
twenty-two Windows NT servers in production, but no 
room to put twenty-two monitors. To consolidate 
access to these devices, a cascaded-K VM switch solu- 
tion from Cybex [17] was procured. 


The primary disadvantage of this solution was 
that, much as was the case with the switchboard sys- 
tem for the serial consoles, it still required that that 
someone go to the data center to use it. A significant 
enhancement to this system, based on the ““VDE/200” 
device from Lightwave Communications [35], is cur- 
rently under evaluation. The expectation is that we 
will be able to use this device to deliver KVM signals 
to the desktops of the systems administration staff; as 
of this writing, we have a single, working control con- 
sole on the same floor as our offices, across the street 
from the data center. 


Still, the high bandwidth requirements of a 
graphical console reduce the chances of getting access 
to the NT server consoles from outside of the two- 
building local campus. One possible alternative is to 
provide some sort of serial console access to an NT 
machine. Pieces of such a solution seem to exist (see 
e.g. [26] ) but it is unclear at this point whether this 
can be made to work with NT in any satisfactory man- 
ner. 


Software Monitoring 


Of course, most (but not all) of these devices 
have some ability to be monitored via software, usu- 
ally within the context of the operating system running 
on the device. Both Solaris and Windows NT have 
SNMP interfaces and a great deal can be done with 
advanced network and system management systems. 
In addition, Solaris provides a fair amount of event 
data through syslog, as does Windows NT through the 
event log. 


Many of the monitoring tasks described in the 
latter portions of this paper could in theory be accom- 
plished through software-based monitoring on many 
of these systems. For example, Sun’s Advanced Sys- 
tem Monitoring [63] package provides driver-level 
access to arbitrary I2C-based [55] monitoring devices, 
including the various monitoring probes that are pro- 
vided on the system board itself. It should be a 
straightforward effort to map these into an extended 
SNMP MIB and accessed through the UC Davis 
SNMP daemon [65] that is used on our Solaris 
machines, or also possibly through CGI scripts run by 
Apache.* Additionally, many newer X86 server sys- 
tems support the new Intel Corporation Advanced 
Configuration and Power Management Interface [29]. 
ACPI — follow-on to the Advanced Power Manage- 
ment Interface (APM) — seems likely to enable the 
collection of a wide variety of system information. 


2Sun also offiers a package called Sun Management Center 
[61], formerly known as SyMON. This is a relative powerful 
package; however, it is unavailable for the UltraAXmp and 
UltraA Xi boards 
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Still, there are three essential difficulties with 
these software-based approaches: 

1. The software work that has to be done tends not 
to be particularly portable from machine to 
machine or from OS to OS. 

2. All these things tend to change at the whim of 
the system vendor. It seems to the author that, 
while by leveraging industry standards such as 
this, one can have a fairly flexible and func- 
tional system in a reasonable amount of time, 
this often comes at the significant cost of sys- 
tem longevity and compatibility among systems 
from overlapping generations. 

3. These capabilities are all very dependent on the 
proper functioning of the hardware and operat- 
ing system being monitored. Generally, this 
means that these interfaces tend to be more use- 
ful for routine monitoring and system configu- 
ration changes than for troubleshooting or fault 
recovery. 


The Harder Stuff 


Still, it was apparent that, even if all these 
‘easy’ tasks were completed, there would remain 
gaps in the available monitoring and control capabili- 
ties, and that closing those gaps was going to be sub- 
stantially harder. 


In-band vs Out-of-Band 


It seemed clear that a monitoring and control 
system would be more useful if it was external to the 
devices being monitored — in other words, if the sys- 
tem was designed to function out-of-band. Malfunc- 
tioning computer and networking equipment cannot be 
counted on to be functional enough to respond to man- 
agement queries and requests. This observation was 
realized by problems we experienced in our early 
attempts to manage the Windows NT servers: it 
seemed that one of the first stages in almost any fail- 
ure mode was to stop talking to the network. 


The in-band nature of all the approaches in the 
previous section (with the exception of the power con- 
trollers) is a major limitation. For example, most ATX 
motherboards by default will only enter standby mode 
when powered up. This is by design and, even though 
this behavior can be changed in the system BIOS, 
there are very good reasons for the default setting — 
one cannot assume that the power will be stable imme- 
diately after it is first restored. Thus, even though the 
power controllers can reset the power to an NT server, 
this is only the first step in restoring the server to full 
function — someone must still push the “‘on”’ button. 


Supporting Heterogeneity 


While many modern computing and communica- 
tions devices incorporate some monitoring and control 
capabilities, the implementation of these tend to be 
highly variable from system to system; monitoring and 
control procedures and software must be developed 
separately for each type of system. If one were to 
deploy, for example, a homogeneous net work 
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consisting of nothing but Sun UltraEnterprise or Com- 
paq ProLiant servers, one could probably design a 
fairly successful monitoring system that depended on 
the specific capabilities of those devices. 


However, in our research network, such an 
approach was not an option. As we started this whole- 
sale network redesign, we had made a commitment to 
acquire most of our computer systems, including both 
SPARC/Solaris and X86/Windows NT servers, by 
integrating OEM parts in our own facility. ‘While 
high-end systems such as those from Sun and Compaq 
are potentially more reliable, they are substantially 
more expensive than those which we build in-house — 
twice as expensive in many cases. Our assessment of 
this trade-off was that, without substantial budget 
increases, the ““more manageable” systems could not 
be deployed in sufficient quantities to meet the com- 
puting demand. As a result, our production systems do 
not generally come with some of the advanced soft- 
ware tools and on-board diagnostics that might come 
with top-of-the-line, commercially integrated systems. 


Not Everything is a Nail 


The data that may be collected from the various 
systems, besides being variable from system to sys- 
tem, is often inadequate for many purposes. For 
example, the Sun UltraAXmp motherboards are able 
to provide a threshold waming when the temperature 
reaches a certain level (recorded by Solaris in syslog). 
If the system gets much warmer, it will shut down. 
While this is useful, it would be more useful if it were 
possible to obtain a steady reading of system tempera- 
ture, that could be routinely recorded into a database 
for trend analysis. 


Also, most systems limit monitoring to what is 
happening inside the system itself, while it is fre- 
quently the case that one would be just as interested in 
what is going on outside the system. For example, 
rack ambient temperature and airflow can be very use- 
ful in identifying pattems in system problems. Many 
systems can report information conceming power sup- 
ply status, but this is usually limited to the DC side of 
the supply; it is rare that a system will provide infor- 
mation concerming the AC side of the supply: current 
and voltage, circuit loading, power sags and spikes, 
etc. Many of our most difficult-to-solve problems 
occur as a result of failures or deficiencies in systems, 
such as the AC power supply, which are controlled by 
other parts of our organization. Obtaining good data 
on those can often be essential to a resolution. 


Finally, much of the instrumentation — both for 
monitoring and control — provided out of the box with 
typical computer systems is designed to be useful only 
to an operator physically present at the system. This 1s, 
for example, the case with the power and reset but- 
tons, as well as the indicator LEDs on the system front 
panel. It is rare that the system manufacturer will pro- 
vide any straightforward mechanism for remote access 
to this native instrumentation. 
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Design 


Thus, in parallel with the efforts described in the 
previous section, an investigation was begun into the 
feasibility of creating a monitoring and control system 
which operates separately from the computer systems 
themselves. 


Design Criteria 


In approaching this problem, several design cri- 
teria were identified: 

e Relatively low cost. It did not seem practical, 
from a budgetary standpoint, for the monitoring 
system to be comparable in cost to the systems 
being monitored. 

e Small size. Even with thirty-five machine 
racks, rack space was highly constrained. If 
these monitoring devices could be made either 
to fit inside the monitored devices, or fit com- 
fortably in the free space at the back of the 
racks, they would be much easier to install. 

e Low power with no special cooling require- 
ments. In particular, it would be most useful if 
it were possible to run these boards during time 
periods when the monitored systems are not 
able to operate. Optimally, the instrumentation 
network could be run for at least a couple of 
days just on battery power. Note that these cri- 
teria imply that the monitoring board cannot be 
powered by the monitored system’s own power 
supply. 

Remotely accessible, preferably using a stan- 
dard interface that is available on virtually any 
computer. Ideally, the system would be accessi- 
ble via a web browser, but access via a charac- 
ter terminal interface would also be acceptable. 
Distributed, modular intelligence — small, sin- 
gle-board intelligent probes, networked 
together and under the control of a higher-level 
supervisory system. Each probe board would be 
dedicated to a single monitored system in order 


to minimize’ cross-dependencies among 
systems.? 
Goals 


As the whole point of this project is monitoring 
and control, it is well to take a look at what it is 
expected that the system will monitor and control, and 
why. 

Monitoring 


e Temperature. Each monitoring board should 
have interfaces to measure four or more tem- 
peratures. One could be measured by a chip on 
the surface of the monitoring board itself, while 
the other three should be in the form of headers 


SThe primary alternatives to this would appear to be (a) to 
have a smaller number of much more powerful systems, 
each monitoring a relatively large number of basic sensors, 
or (b) to have a network of individual, intelligent sensors. 
For a variety of reasons, these were rejected fairly early on 
in the project. 
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that will allow the connection of remote sen- 
sors. These sensors could be placed on CPU 
heat sinks, disk drives, outside of the system, 
etc. The author views temperature as the single 
most important indicator of system health. If 
anything is seriously wrong with a system, 
chances are good that the temperature will 
either go up or go down. 

DC Power supply voltages. The monitoring 
board should have headers that can be con- 
nected to contacts on the motherboard which 
carry +12V, +5V and +3.3V. Drift or instability 
in these values can indicate a weak power sup- 
ply. 

Fan rotation. Many fans today have a tachome- 
ter lead. On each rotation of the fan, this lead is 
shorted to ground a number of times (four is 
typical) Feeding a voltage to the tachometer 
wire through a pull-up resistor will result in a 
square wave with a frequency that is directly 
proportional to the fan’s RPM. Typically a 
computer system board will provide this rate 
counter functionality itself, but it is straightfor- 
ward to tap into this signal and also do the 
counting on another board. The monitor board 
should be able to keep track of at least two 
fans. 

LED states. When confronted with a malfunc- 
tioning system, many people will take a look at 
the hard drive activity LEDs on the front panel 
as a quick check to see if there is any disk 
activity. If the light stays on solid, it can be an 
indication of a hung SCSI or IDE bus. If it is 
off solid, this can be an indication that the oper- 
ating system is not running any tasks. If it is 
blinking rapidly, it would appear that something 
is going on — often, by the blink rate, just what 
is happening on the system can be deduced. 
The monitoring board should have connections 
to monitor the state of up to three LEDs. It 
should be able to report the current on/off state 
as well as a current count of off-to-on transi- 
tions. 

AC current. There are two places at which it 
would potentially be useful to monitor AC cur- 
rent: At each circuit and at each device. At the 
circuit level, a current reading will give one a 
sense of how close a circuit is coming to capac- 
ity, and how much more can be added to it 
without concern. Monitored over time, one 
potentially can detect, for example, that a new 
device has been added to a circuit, or that one 
was added during the night and then removed 
(those darn cleaning crews!). With a little more 
work, it would also be possible to monitor volt- 
ages, and look for spikes and overall power 
quality. At the device level, one could addi- 
tionally detect, on a per-device basis, sudden 
increases and decreases in current demand that 
could be indicative of a number of system level 
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events such as impending failure, shutdown or 
reboot. 


The monitoring board should be able to moni- 
tor a single AC current at at the device level. 
We determined that the task of monitoring AC 
at the circuit level had a number of complexi- 
ties that made it more appropriate to leave to a 
later project. 

RS-232 serial interface. The monitoring board 
should have a minimum of two RS-232 serial 
interfaces in addition to some sort of network 
interface. In some cases, such as with the CMD 
RAID controller, useful tnformation such as the 
state of a RAID set is available only through a 
serial interface. It also may prove practical to 
interface these devices to the serial console 
ports on the SPARCservers and network 
switches, but with the serial console network 
discussed above, this is left for another day. 
One serial interface would be used for these 
applications, while the other would be left 
available as a debugging and local control inter- 
face. 

Generic digital inputs. There should be four or 
more generic digital inputs to account for plat- 
form specific capabilities, for example to sense 
alarm conditions that are provided by some 
chassis and system boards. At least four such 
inputs are known to be required to moinitor our 
RAID subsystems. One possible use for such an 
input would be to create a sort of “watchdog 
timer”. One might write a daemon or NT ser- 
vice which toggles the state of one pin on the 
parallel port every five seconds or so. The mon- 
itor board could watch for that change and, if it 
does not occur for twenty seconds or so, set a 
flag which could be picked up and be reported 
in an alert. In embedded systems, a timeout on 
the watchdog timer is usually programmed to 
result in a system reset; with some caution, this 
approach could be taken for the monitored sys- 
tem via the relay outputs mentioned below. 
Generic analog inputs. There should be two or 
more generic analog inputs, again to measure 
platform-specific values, which could be addi- 
tional temperatures, airflow, humidity, etc. 


Control 


Relays. There should be two relays on the mon- 
itoring board, each of which can be used to 
close a switch header on a motherboard, such 
as those for the reset or power on/off functions. 
These should be specifically designed to not 
close except under tightly controlled circum- 
stances. Of special concern is during monitor- 
ing-board power-up and when the relays are 
connected or disconnected from the monitored 
system; the monitor board should never cause 
the monitored system to shut down or reboot. 

RS-232 serial. The same interfaces that were 
described under “‘monitoring” can clearly be 
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used in a control application; for example, a 
command could be added to shut down a RAID 
array before powering off a server. 


Taking a closer look 


With these goals and criteria in mind, several 
candidate technologies were examined. But first, let us 
note what is not available. 


The market, as far as could be determined, was 
barren of products that directly met the goals and cri- 
teria listed above. Generally, the industry seemed to 
be taking the view that computers could monitor 
themselves just fine. Several computer system vendors 
had system and network management solutions. But, 
almost without fail, these solutions depended on the 
existence of a homogeneous collection of reasonably 
high-end systems, and/or they functioned in-band with 
the operating system and primary data network. 
Although many vendors were consulted, and many 
had suggestions as to how to approach the task, not 
one claimed to have a solution on the shelf. 


Following is an outline of several technologies 
which were considered for this project. 


PC-104 Single-board Computers 
Features 


PC-104 is a standard form factor for X86 PC/AT 
computers, using 4” square boards and stacking pin- 
and-socket connectors to carry the ISA bus. A wide 
variety of peripheral and I/O boards are available. 


Advantages 


The advantage of PC-104 is the use of a stan- 
dard, familiar PC architecture. 


Disadvantages 


High cost. A fully-functional system can cost in 
excess of $1000. Proprietary aspects often complicate 
software development. Generic sensor interfaces 
require external signal conditioning circuitry. 


Vendors 

Ampro [5], Advantech [2], Technoland. 
Single-board Microcontrollers 
Features 


Small boards configured with standard micro- 
controllers. Usually provide a variety of digital and 
analog interfaces, not normally designed for expan- 
sion. 

Advantages 

Moderate cost, well-documented and less propri- 
etary. 
Disadvantages 


Limited mix of sensor interfaces often restrictive. 
Generic sensor interfaces require external signal con- 
ditioning circuitry. 

Vendors 


EMAC, Inc. [36], Z-World [71], R.L.C. Enter- 
prises [58]. 
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Industrial Control PLC Networks 
Features 


Programmable Logic Controllers are pro- 
grammed with basic state machine logic. Networks of 
individual sensors and controls are connected back to 
the PLC. 


Advantages 


Very flexible; wide variety of controllers and 
sensors available. 


Disadvantages 


PLC architecture is very limiting. individual 
sensors and controls are large and relatively expen- 
Sive. 


Vendors 
Advantech [2], Omega Engineering [53]. 
Custom-designed microcontroller boards 


For better or worse, this is the one approach that 
seemed to be capable of addressing the bulk of the 
stated goals and criteria. Among the more popular 
microcontrollers, PIC microcontrollers from Micro- 
chip Technology [41] are tremendously easy to work 
with, and these were selected for the first attempts at 
custom design. 


The PIC microcontrollers are programmable in 
their own quirky assembler language or in any of sev- 
eral higher-level languages, most notably C. Pro- 
grams are written into the PICs with low-cost device 
programmers. The model which saw the most use in 
this project was the PIC16C76 [42], a 20MHz, 28-pin 
device with five analog inputs, 8KB of program mem- 
ory, three timers and a real programmable serial port. 
These chips cost between $5 and $10 depending on 
quantity, and the minimal parts one needs to add to 
have a functional controller system (e.g. crystal, volt- 
age regulator, a few capacitors) cost maybe another 
$10. On a board based on a PIC controller, one can 
easily include all the I/O interface circuitry that one 
would have needed to design for all the other solutions 
anyway. Several companies, e.g. MicroEngineering 
Labs [43] sell prototyping boards for the PIC for $10 
or so — with these one can have a functional controller 
board with a couple of hours of soldering. 


This custom design solution, although perhaps 
not the the one which intuition would select, proved to 
be by far the most straightforwardly malleable, to be 
the most cost effective — even when the cost of devel- 
opment tools 1s accounted for — and the most free of 
proprietary encumbrances. With this approach, 
promising, concrete results were obtained in a very 
reasonable timeframe. 


A Word About Networking 
Physical layer 

Three technologies were considered as a choice 
for the physical layer: RS-485, Ethernet, and CAN. 


This in fact tuned out to be a fairly straightforward 
choice, however. 
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RS-485, which is also basis for differential SCSI 
and many short-haul modems, is very simple in 
design. It transmits a differential signal over common 
copper twisted pair wire, one pair for half-duplex and 
two pairs for full-duplex. A ground retum wire is usu- 
ally present as well, so an RS-485 network cable will 
most likely have either three or five conductors. Data 
can be transmitted over such cables at relatively high 
rates, over reasonable distances, and with a great deal 
of noise immunity. RS-485 is usable in a bus topology, 
allowing the connection of dozens of devices to a sin- 
gle length of three- or five-wire cable. At distances 
likely to be experienced in a data center, RS-485 can 
easily approach speeds of 115.2 Kbps, while over 
short distances — as found in SCSI busses — it can sup- 
port speeds as high as 10Mbps or more. 


Upper Layers 


While the selection of a physical layer was fairly 
straightforward, there were no obvious choices for the 
higher-level protocol. TCP/IP was out of the question 
— the software required to implement TCP/IP was 
much too complex to deal with in an 8-bit microcon- 
troller. There appeared to be no “standard”’ protocol 
available; there were several proprietary protocols 
such as Echelon’s LonWorks [24], but it was also very 
common for embedded systems developers to ‘roll 
their own”’ protocol. After an evaluation of the avail- 
able options, the latter approach seemed the most 
promising. 

A Networking Architecture 


The outlines of an architecture for the network- 
ing of the monitoring boards was developed, although 
never used for reasons which will become clear. For 
the benefit of those who may wish to take this less 
proprietary approach, this architecture is described 
here. 


Master-slave model 


In order to reduce the complexity of the code 
which must run on the microcontrollers, and to elimi- 
nate the need to handle collisions, we decided that the 
microcontrollers would speak only when spoken to. 
Each RS-485 string would have a single master node 
which would direct the microcontrollers to retum data 
or carry out a control task. In idle time between 
requests, the controller would constantly be taking 
measurements in order to reduce the polling latency. 


RTOS with simple dispatch table 


Each microcontroller would run a relatively sim- 
ple Real-Time Operating System, which would handle 
task scheduling, interrupt, timer and queue manage- 
ment, as well as resource (e.g. memory) allocation. As 
requests come in off the network, a process running 
under the RTOS would scan request headers and select 
out only traffic for that controller. As each such packet 
comes in, a command token in the packet and any 
needed data would be parsed out and inserted in an 
appropriate job queue. A process running under the 
RTOS, until this point blocked in wait for the queue to 
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fill, would be woken up and given an opportunity to 
act on the command. 


Commands would be classed as synchronous or 
asynchronous. In the case of a synchronous command, 
the master node would do nothing — and the network 
would be quiet — until the slave node retumed the 
results; a timeout would be set so that a dead node 
would not hang the network. In the case of an asyn- 
chronous command, the master node would wait for a 
receipt acknowledgement but would then go on with 
other work, giving the slave node time to finish the 
command. When the command was complete, the 
slave node would buffer the results and wait for 
another poll from the master node before attempting to 
return the results. A request to transmit previously 
requested results would of course be a synchronous 
command, requiring immediate response (or negative 
acknowledgement) If the results are never requested, 
the slave node would discard the results when the 
buffer was needed again. 


RS-485 Network 
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Figure 1: Instrumentation network diagram. 


On the Wire 


Request packets would contain a preamble, a 
one-byte target address, a one-byte command token, 
one-byte request sequence number, an optional data 
field, a checksum and a trailer. With a single master 
node, a source address would not be required in a 
request packet. Response packets would have a 
slightly different preamble, the source address and 
response code, followed again by the sequence num- 
ber, optional data, checksum and trailer. Request pack- 
ets would need to be acknowledged by the microcon- 
troller within a set time window, but response packets 
would not require acknowledgement; instead, the mas- 
ter node would simply repeat the request if required. 


Start-up node discovery would be done by a sim- 
ple search through all possible addresses — basic 
RS-485 networks support only 32 devices and this dis- 
covery can be completed in only a few seconds.‘ 


Master Node 


It 1s expected that the master node would be a 
Unix-type machine, probably Linux or BSD. On this 


4This is an electrical limitation related to the input 
impedance of the transceivers; high-impedance transceivers 
are available which will allow up to 128 devices to be at- 
tached, but the use of these seems unnecessary in this appli- 
cation 
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master node, the protocol would be implemented as 
user-space executables which could be called by CGI 
scripts from the Apache web server. The service 
requests and results could be delivered from and to the 
user via the http protocol. Automatic cron jobs could 
run on the master node to make routine, periodic 
requests for data that would then be stored in a 
database. 


Disadvantages 


There are some disadvantages of this system; it 
does not, for example, provide a mechanism for the 
microcontroller to report alerts or interrupts back to 
the master node. A non-responsive or ill-behaved 
slave node would need to be flagged as a potential 
problem, and an alert generated through the network 
management system. Also, the number of devices 
hanging off each serial port would also be limited by 
how many could successfully be polled without using 
up all the available bandwidth with overhead process- 
ing. 
em Ware 

This architecture was not implemented, however, 
because a source, emWare [25], was identified for a 
product that did essentially the same thing. emWare’s 
product, EMIT (Embedded Micro Internetworking 
Technology), provides a networking protocol, an http 
gateway, skeleton source code for several microcon- 
trollers, and Java Beans that can be used to create a 
graphical interface to the networked controllers. It was 
estimated that that this product would take at least six 
to twelve months off the time to develop the instru- 
mentation software, and a much more functional end 
product would result than would have otherwise. 


First Working Prototype 


Ultimately a wire-wrapped, PIC-based prototype 
was designed and built. This prototype did most of 
what was required, including running the EMIT soft- 
ware and talking to a web browser via the Java Beans. 
The browser window would display the constantly- 
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updated board temperature, display counts of the HDD 
activity lights, and a Java-based button could be 
depressed to initiate a hardware reset on a Windows 
NT server. Finally, an affordable and buildable solu- 
tion was in sight. This was in the fall of 1999, and this 
board was the one that was described to some LISA 
participants in a Work In Progress session last year. 


After completing this prototype, however, it 
became clear that the PIC had a severe shortcoming: 
The PIC simply did not have enough data memory, 
and it had no external memory interface. The. most 
serious problem resulting from this was in interfacing 
with the CMD RAID controller’s serial port. Although 
the RAID controller had a mode for talking to such 
automated devices, use of that mode entailed the abil- 
ity to buffer records that were comparable in size to 
the PIC’s entire memory. Thus, a different microcon- 
troller was going to be needed. 


Atmel AVR 


After a review of the alternatives, Atmel’s [8] 
AVR architecture was selected, specifically the 
ATmegal03. The AVR was still an 8-bit microcon- 
troller and was still inexpensive, but the instruction set 
was much more powerful than the PIC’s and it had an 
external memory interface. Over the next few months, 
a fairly complete board was designed around on this 
part, while many design details were tested on low- 
cost development boards that are available from 
Atmel. At around the end of Spring of this year, the 
design was nearly complete, and attention was given 
to the task of procuring the parts that would be 
required. Especially in a small project such as this, it 
is generally a good idea to have the parts in hand 
before the boards are made, so that adjustments can be 
made in the event that some parts are not available. 


The Parts Shortage of 2000 
One does not normally, however, expect the 


availability problem to be as severe as was experi- 
enced in this case. Parts were so hard to come by this 





Figure 2: Conceptual drawing of monitor board. 


50 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Drzyzgula 


past Summer, that it was headline news every week in 
publications such as EE Times and Electronic Buyer’s 
News (EBN). Pretty much anything in a surface- 
mount package, anything with memory in it, or any- 
thing that was used with anything that was surface 
mount and/or used memory — especially flash mem- 
ory, was simply not available. EBN ran a story about 
an OEM CEO who raided his employees’ Palm orga- 
nizers just to get the flash memory [52] ; see also [69] 
and [68]. In the case at hand, about half the parts on 
the board — which was designed using surface-mount 
technology — were unavailable, most notably the 
Atmel microcontroller itself. Atmel reportedly was so 
far behind that they had ceased taking orders, and 
Atmel distributors were quoting lead times as long as 
ten months. There was no way around it, it was either 
shelve the whole thing for a while or go back to the 
drawing board. 


Yet Another microcontroller 


Thus, the search began for yet another microcon- 
troller. This time, the search was limited to devices 
which could (a) use as much of the existing hardware 
design as possible, (b) were available for immediate 
purchase and (c) for which there existed a port of the 
emWare code. Given the scarcity of parts, this tumed 
out to be a pretty short list; very few microcontrollers 
of any sort were obtainable at that point. In the end, a 
supply of Dallas Semiconductor DS 80C320 chips was 
located. The DS 80C320 is sort of a souped-up, high- 
speed 8051-architecture microcontroller with no inter- 
nal program storage. They also have relatively little in 
the way of of built-in peripherals, so the search had to 
be expanded to include several new parts, such as an 
external analog-to-digital converter and EEPROM 
memory. Parts in hand, work was begun to design the 
board one more time. However, this time around, it 
was going to have to be larger — the perimeter of a 
CD-ROM drive rather than of a floppy drive as origi- 
nally planned — and would have a higher power con- 
sumption than desired. But something had to give, and 
at least work was still progressing. 


Working Design 


Overview 


Almost all of the parts used in this board will be 
‘‘through-hole”’ as opposed to ‘‘surface-mount”’. The 
board itself will be designed in four layers (two signal 
layers, a ground layer and a power layer), with the 
outside dimensions about the same as that of a CD- 
ROM drive. Mounting holes will be drilled near the 
edge in such a way as to make it straightforward to 
mount the board, using angle brackets, in a standard 5 
1/4” drive bay. The plan is to mount this device in a 
free hard drive bay inside the server device. The 
instrumentation network and power supply for the 
board will enter the system through one of the I/O slot 
hanger brackets, and bundles of sensor wires will be 
connected from various locations within the server to 
connectors on the back of the monitor board. 
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A conceptual drawing of the board design is pro- 
vided in Figure 2. 


Power 


The boards are designed to operate from a +24V 
power supply. The relatively high DC voltage is used 
for two reasons: First, some devices, such as the 
4-20mA current loop, need the 24V to operate. Sec- 
ond, the plan is to daisy-chain the power supply along- 
side the RS-485 network. If the boards were designed 
to run from a +5V power supply (voltages will need to 
be converted on-board in any event), the supply cur- 
rent would have to be higher; this would have a num- 
ber of implications, including the use of heavier wires 
in the daisy-chain hamess. The 24V power is con- 
verted to +5V, -12V and +12V on-board using off-the- 
shelf DC-to-DC converters. 


Connectors 


All the run-time I/O connectors will be on one of 
the short edges of the board. The design calls for three 
connectors. One rectangular nylon connector will 
carry connections for power input, the RS-485 net- 
work and the 4-20mA current loop. 


The other two connectors are both 26-pin high- 
density D-sub connectors. One will carry most of the 
analog connections, while the other will have the 
serial ports, relays, counters and other digital connec- 
tions. 


There will also be small header connectors on 
the surface of the board for the attachment of JTAG 
cables, for the programming of the two programmable 
logic parts used in the design. 


Components 


In this design, there are about twenty or so inte- 
grated circuits, and about two or three dozen other 
parts. Some of the more interesting parts are described 
here; refer to Figure 3 to see how these parts work 
together. 


Dallas Semiconductor DS80C320 


The Dallas Semiconductor DS80C320 [21] is a 
clone of the Intel 80C32 [30]. Dallas’ part uses a new 
core design, runs faster than the old Intel part (up to 
33MHz), and does more in each clock cycle than the 
traditional 8051 designs. The Dallas part also has an 
extra serial port, extra timers and a few extra external 
interrupt pins. This is actually about as nice as 
8051-architecture chips get. 


Waferscale Integration PSD9]3F2 


Wa ferscale [67] makes a line of highly-integrated 
parts that they call “Programmable System Devices”’, 
or “PSDs”; the current design uses the PSD913F2 
[66], which provides 160KB of flash memory in 
twelve sectors, 2KB of SRAM and a 57-input/19-out- 
put simple PLD. The PSD provides all the program 
storage for the board, some of the data storage, and 
handles all the address decode work needed to gener- 
ate chip selects for the other ICs on the board. 
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National Semiconductor LM12458 


National calls the LM12458 [48] a “data acquisi- 
tion system”’. It is a highly programmable analog-to- 
digital converter, with eight inputs. The conversion 
circuitry can give a result with twelve bits of preci- 
sion, and it can buffer several results to take some pro- 
cessing load off the microcontroller. It can also be pro- 
grammed with threshold values, which allows for the 
generation of an interrupt if a sensor drifts out of spec- 
ification. 


Altera MAX7064S 


The Altera MAX7064S [4] is a 64-macrocell 
CPLD, (Complex Programmable Logic Device). The 
purpose of this part here is to handle the counting of 
the three HDD activity LED interfaces. The three LED 
headers on the monitored system’s motherboard will 
be connected (via optoisolators and buffers) to input 
pins on the CPLD. The CPLD increments an internal 
counter every time one of the counter input pins tran- 
sitions from low to high. It communicates to the 
microcontroller though a parallel interface; the micro- 
controller thinks that the CPLD 1s just three bytes of 
memory. 


Dallas Semiconductor DS1780 


The Dallas DS1780 [20] is a system monitor 
chip, designed for use on PC motherboards. It 1s capa- 
ble of monitoring the surface temperature at the chip 
itself, as well as two fan speeds, several power supply 
voltages, and various other parameters. It has a regis- 
ter that latches the state of five external pins on power- 
up; this is designed to read an identifier off a Pentium 
processor, but here it is used with a DIP switch to 
latch the board’s address on the RS-485 network. 
Finally, the DS1780 has a system reset circuit, which 
will keep all the chips on the board in a reset state 
until the power supply has stabilized. 


STmicroelectronics M24C32 


The M24C32 [59] is a byte-addressable, 32 Kbit 
serial EEPROM. Although there is plenty of flash and 
SRAM memory elsewhere on the board, neither of 
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these are particularly good at storing small amounts of 
dynamic data which must survive a system power 
cycle. This device can be used to store various setup, 
configuration and calibration parameters such as the 
monitor board’s monitored system name. It can also 
occasionally be used to log various critical events that 
one might need to retrieve after the system was power 
cycled. 


Philips Semiconductor PCF8584 


Since the DS 80C320 does not have an I2C inter- 
face, and both the DS1780 and the serial EEPROM 
required one, the design calls for a Philips PCF8584 
[54], an I2C bus controller, to do this job. 


Dallas Semiconductor DS1687 


The Dallas DS1687 [19] is a battery-backed real 
time clock (RTC). It will be used by the board both to 
keep track of the time and to generate a square wave 
input to the microcontroller which will provide a regu- 
lar system ‘“‘tick”’ for the RTOS, that is used to sched- 
ule various housekeeping activities. This part also pro- 
vides 256 bytes of NVRAM. 


Burr-Brown RCV420 


The Burr-Brown RCV420 [10] is a 4-20mA cur- 
rent loop receiver. 4-20mA current loops work by sup- 
plying around 24V to a pair of wires which are carried 
out to a remote sensor. The sensor is powered off the 
loop wires themselves, and communicate back to the 
receiver by regulating the loop current to be a value in 
the range of four to twenty milliamps which, net of the 
4mA base, is proportional to the value the sensor 
wishes to report. Current loops can be run for long dis- 
tances, require only two wires to carry both power and 
signal, and have excellent noise immunity. 


The RCV420 senses the loop current and outputs 
a voltage proportional to the reported current and in 
the range of 0-5V, which is what most analog to digital 
converters are designed to measure. In this board, the 
RCV420’s output will be connected to the LM12458. 
The RCV420 will be used to interface to a remote AC 
current sensor. 
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Figure 3: Block diagram of monitor board functional components. 
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Sensors 
Temperature Sensors 


In addition to the single temperature measured 
by the DS! 780, three remote temperature sensors will 
be supported. The remote sensors will use the National 
Semiconductor LM20 [49], a tiny, 2mm x 1.25mm x 
Imm surface-mount part. 


AC Current Sensors 


Although this was not anticipated, the AC cur- 
rent sensor interface turned out to be one of the hard- 
est parts of the entire design to get right. 


It 1s quite easy to come by remote current sensors 
which transmit information via 4-20mA current loop — 
this is a standard type of part used in industrial process 
instrumentation applications. However, while the 
specifications of hundreds of toroidal-transformer and 
Hall current sensors from several different vendors 
were reviewed, none were identified which simultane- 
ously met the specifications and goals for size (to be 
able to put one in a regular outlet box, or inside the 
power controller), frequency response (60Hz), sensing 
range (1-20A) and cost (<$50). In addition, for many 
industrial current sensors, the translation between the 
AC current and a single value to be transmitted on the 
4-20mA current loop is not always well-defined or 
particularly sophisticated. A sample of one such sen- 
sor was disassembled and inspected, and determined 
to use a particularly simple rectifier and linear integra- 
tor circuit which would, at best, give only an “indica- 
tion” of the current level, not something that could be 
reliably mapped to an RMS value. 


This is particularly an issue with the kinds of 
loads presented by computer equipment. Unlike resis- 
tive loads, which will usually have AC current profiles 
which look pretty much just like the voltage profile 
(1.e., sine waves), the switching power supplies used 
in computer equipment consume AC current in short 
bursts. This sort of signal is said to have a high “‘crest 
factor’’; this presents special challenges in the mea- 
surement of an absolute AC current level — see [33]. 
Many industrial current sensors are designed under the 
assumption of either resistive or inductive (motor) 
loads; these assumptions do not necessarily translate 
well into this application. Thus, it is not really fair to 
say that the device mentioned in the previous para- 
graph was poorly-designed; rather, it 1s more that it 
just was designed for a different application. Errors in 
current values reported by simple averaging circuits 
for signals with high crest factors can be as high as 
50% or more. 


Candidate design 


In the current sensor design developed for this 
project, the transducer is a small, toroidal current 
transformer made by Coilcraft [15]. This is an inex- 
pensive ($5 or so) part which is just a little more than 
ten cubic centimeters in volume. The current output of 
the CS60 is dropped across a small resistance, and the 
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resulting voltage is input into an ““RMS to DC Con- 
verter” — the AD737 [6] from Analog Devices. To 
translate the output of the AD737 into a 4-20mA cur- 
rent signal, a Burr-Brown XTRI1O1 [11] current loop 
loop transmitter 1s used. 


While, as of this writing, this circuit design is not 
complete, it should result in a relatively small pack- 
age, as current sensors go, and should cost $50 or less 
to build. It is believed that the primary way to improve 
on this design is by use of a high-speed analog-to-digi- 
tal converter and a digital signal processor chip, which 
would be considerably more complex and costly. 


While the requirement for a 4-20mA current loop 
receiver adds complexity to the overall design, the 
availability of such an interface on the monitoring 
board, because of the wealth of sensors with 4-20mA 
outputs available for industrial process instrumenta- 
tion systems, gives this board a great deal of flexibility 
in being able to handle new and unanticipated applica- 
tions. 


RTOS 


While, at this point, emWare’s EMIT software 
appears to be quite suitable for this project, the skele- 
ton microcontroller code provided has a number of 
shortcomings. It generally will work, but it is not par- 
ticularly modular. One would like to see a structure for 
building modular, cooperative processes calling 
library routines which implement a well-defined API, 
but the emWare code is more like a giant, monolithic 
event loop into which one inserts one’s own code. It 1s 
designed sort of like an RTOS, but a number of essen- 
tial RTOS services are missing, or blurred into the 
main event loop. 


Thus, the plan is to break that code up — much of 
it may have to be substantially rewritten — and re- 
implement it as a number of modular tasks under the 
control of an actual RTOS. This is somewhat less 
daunting than it may seem, because the existing code 
does at least provide a good, working example, and a 
large proportion of the emWare code is not useful in 
this application (e.g. modem support) or is not rele- 
vant in the context of an RTOS. 


The working plan has been to use Jean 
Labrosse’s uC/OS-II as the RTOS. One purchases the 
source code to uC/OS-II by purchasing Labrosse’s 
book on the product. What comes with the book is 
only licensed for non-commercial use and is unsup- 
ported in the sense that one has to purchase upgrades 
and bug fixes. 


In researching references for this paper, however, 
the author leamed of a new, fully free RTOS that had 
just been released in mid-1999. The PROC [50] Real- 
Time Kemel, written by Jan Erik Nilsen of Nilsen 
Elektronikk, is a very straightforward RTOS kemel 
which has been ported to several microcontrollers and 
would seem to be fully capable of meeting the require- 
ments for this project. 
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Status 


Hardware 


As of this writing, the schematics for the board 
design are largely complete except for some of the 
analog interfaces. The next major step is to develop 
the board layout, which amounts to drawing where the 
chips go on the board and how all the traces get from 
one chip to the other. When that is complete, the 
design will be taken to the board fabrication house to 
have one panel of printed circuits made. A few of 
these will be populated with parts and tested by hand. 
If the boards work as planned, further assembly will 
be moved to a contract manufacturer. 


There are some additional boards which have to 
be made, which are various stages of completion: 

e Since most PCs do not have an RS-485 port, an 
RS-232 to RS-485 converter board [22] was 
designed. This board works by simply attaching 
an RS-232 transceiver and an RS-485 
transceiver back to back. One panel’s worth of 
these boards have been made, and the samples 
that have assembled have worked exactly as 
required. Thus, this board 1s largely complete. 
There was only one part for this design which 
was not available in a through-hole form — the 
Dallas DS1780 — and no similar system moni- 
tor chip seemed to be available in through-hole 
from any other vendor. Thus, rather than have a 
single surface-mount part on the board, a small 
adapter board was designed and enough were 
manufactured to last through the first produc- 
tion run of the monitoring board. 

Boards on which to assemble the remote tem- 
perature and AC sensors will also be required; 
these designs are partially complete. 


Software 


One of the hardest parts of doing this work 1s 
that so little of the software can be written and tested 
before the hardware design is done. Thus, while an 
architecture for the software has been developed at a 
fairly high level, and both the emWare/EMIT and 
uC/OS-II software have been tested on actual hard- 
ware, little has been done of the down-and-dirty work 
of banging out the code to make this thing all work. 


Availability 


Neither the author nor his employer have any 
proprietary interests in the portions that have been 
developed by us or at our behest. This includes the 
general system architecture, the hardware designs, 
including all printed circuit board schematics and lay- 
outs. As portions of this work are completed, it is 
intended that these will be made available on the Inter- 
net — of course with no warranty or support. Two por- 
tions of the design, the RS232-to-RS485 adapter 
design [22] and the Null-modem adapter design [23] 
are available in this way, and others should be avail- 
able soon, as time allows. This paper is in part 


54 


Drzyzgula 


intended to serve a similar purpose. If the reader has a 
specific interest in some portion of what has been 
done here that has not yet been made available, please 
let the author know. 


However, since many portions of the project 
make use of proprietary software tools, there may be 
some gaps in what can be released. For example, to be 
able to modify, compile and run the software that will 
be written for the monitoring board, a potential user is 
likely to have to obtain licenses from Tasking [64] for 
the C compiler, and emWare [25] for the EMIT SDK. 
Both the Waferscale and Altera programmable logic 
parts were selected in part because their manufacturers 
provide excellent free development tools. 


This being said, one secondary goal in this pro- 
ject has been to avoid the dependence of the hardware 
components on commercial software packages. It 
should be quite feasible to build a few of these moni- 
toring boards and write completely different program- 
ming for them. Implementations of the services pro- 
vided by the commercial packages could likely be 
written from scratch if one is sufficiently determined. 


It should also be noted that all these printed cir- 
cuit board designs are being prepared with the Eagle 
layout editor, from CadSoft [12]. While these files 
will not be importable into competing design pro- 
grams, they will be readable and printable with Cad- 
Soft’s freeware “lite” version of Eagle, which also 
serves as a viewer. However, one would not be able to 
modify the designs with the free version because they 
are larger and have more layers than the free version 
will allow to be edited. The professional version of 
Eagle is, however, quite reasonably priced compared 
to many other EDA packages. Also, the “Gerbers”’ 
will be made available — Gerbers are the native format 
for the Gerber photoplotters, and have become the lin- 
gua franca of the PCB fabrication industry. 


Project future 


Looking toward the future, the author believes 
that an optimal outcome would be that this idea — of a 
instrumentation network, designed in response to 
requirements specific to a data center environment and 
which operates quite independently from the produc- 
tion computer and communications systems — will 
catch on, and we will ultimately see commercial prod- 
ucts along these lines, complete with “‘connection”’ or 
“probe” kits for various commercial and generic com- 
puter systems, much in the way that KVM switches 
are sold today. The author hopes that, were this to 
come to pass, the controller modules themselves will 
be designed in an open and standard manner, so that 
users of such equipment will be able to expand the 
capabilities both of the hardware and of the software. 


Toward this end, it seems that one significant 
contribution that could be made by this project would 
be to begin to define standards for how such an instru- 
mentation network should operate. In that regard, a 
starting outline is offered here: 
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Sensor Interfaces 


Some of the hardest parts of this hardware design 
are the interfaces between the microcontroller and the 
sensors or controls. While 4-20mA interfaces will suf- 
fice for many sorts of sensors, for others they may not 
be as practical or cost-effective. 


Since temperature sensors would be a frequent 
choice in this application, it would make sense to opti- 
mize an interface just for these. Temperature sensors 
vary considerably in their electrical interfaces. For 
example, thermistors are devices which vary in resis- 
tance over temperature, thermocouples generate tiny, 
temperature-depende nt voltages with response curves 
that are quite dependent on the metals used, while 
some IC sensors, having pre-linearized voltage out- 
puts, vary by 10mV per degree. The interface require- 
ments for these three types of sensors are quite differ- 
ent. The differences are, however, sufficiently small 
that it would be fairly simple to specify one or two 
standard temperature sensor interfaces for a computer 
monitoring board which could support the connection 
of any of these types of sensors through a simple con- 
ditioning circuit. A four-pin connector, for example, 
carrying +5V supply, supply ground, and a differential 
analog signal retum, or a five-pin connector with 
+]2V, -12V, supply neutral and the differential return, 
would likely work quite well for such an application. 
By defining these interfaces, the board design 
becomes more straightforward, and standard sensor 
designs, covering a wide range of functionality, could 
be defined and mass-produced. 


Controller Hardware Architecture 


While this may not be the most practical area for 
standardization, the creation of one or two reference 
designs can be very useful in creating a standard 
which can be widely implemented. An important con- 
sideration here will be the selection of an architectural 
level for the microcontroller — 8-bit, 16-bit or 32-bit. 
This choice will have a great impact on many other 
design decisions, such as the RTOS or the networking 
protocol. It will also determine much with regard to 
the overall cost and complexity of the design. The 
author’s personal assessment is that this application 
calls for an 8-bit architecture, although a 16-bit pro- 
cessor, while possibly overkill, might be considered if 
strongly indicated by other considerations. 32-bit 
architectures would appear to be excessive at best. 


Controller RTOS 


The RTOS, or Real-Time Operating System, that 
would run on the monitor boards is something that 
would be very useful to standardize. An important cri- 
teria for RTOS selection is likely to be the openness, 
or “freeness” of the RTOS architecture and imple- 
mentation both for commercial and non-commercial 
use. There exist several RTOSs which are “‘free’’ in 
the sense that one can download the source and tinker 
with it, but which are otherwise restricted in ways 
which limit their usefulness for an open, collaborative 
project. uC/OS-II [40], for example, has this problem. 
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There are also several RTOSs which are fully free in 
the sense required, but which are designed only for 
32-bit or wider processors. Some examples of these 
are RTEMS [51], eCos [57] and the various embedded 
(although in some cases not fully Real-Time) Linux 
versions (Embedix [37], uClinux [38] and Hard Hat 
Linux [45] ). 


Given an assessment that 32-bit architectures are 
uncalled for here, this would imply a need for a free 
RTOS targeted to smaller architectures. Unfortu- 
nately, these seem to be somewhat less common. 
Three RTOSs meeting this criteria have been identi- 
fied by the author: Uros Platise’s UROS®, Nilsen Elek- 
tronikk’s PROC [50], and Kate Alhola’s Katix (3]. 
Another, Barry Kauler’s Screem [31] may also be suf- 
ficiently free for such a project. PROC appears to be 
the most interesting, complete and portable of these. 


Driver Interface 


Within the context of a selected RTOS, the con- 
cept of a sensor driver module could be defined, lead- 
ing to significant opportunities for code reuse. 


Networking Protocols 


A networking protocol would of course be cru- 
cial. While searching for information in support of the 
research for writing this paper, it became clear that the 
state of — or at least the state of information on — 
higher-level protocols for CAN has improved consid- 
erably since the early stages of this project. It seems 
likely that CAN, together with some reasonably open 
higher-level protocol, would make an excellent choice 
for. a networking protocol in a standard for this kind of 
instrumentation system. 


The difficulty with CAN is that, like Ethemet 
and to a lesser extent RS-485, CAN is a specification 
at the physical and data link layers of the OSI stack. 
These layers typically are implemented in silicon, 
either as an on-board peripheral in a microcontroller, 
Or as an external interface chip. This means that, in 
order to be useful, a CAN design must include an 
““HLP”, or Higher Level Protocol. Unfortunately, 
there are literally dozens of HLPs to pick from, and 
most are proprietary, incompatible with one another 
and highly application-specific. This has also resulted 
in a great deal of fragmentation in the CAN market, to 
the point that it is not clear to this author that any 
complete, fully open, free software implementation of 
a CAN HLP stack exists. For more information on 
CAN and the tangle of HLPs, see [34], [39], and [32]. 


Nonetheless, this must be moderated by the prac- 
ticality of implementing such a protocol within the 
context of the selected microcontroller and RTOS. The 
PROC Real Time Kemel includes an architecture for a 
basic multi-processor message-passing interface 
which may prove sufficient for an application such as 
this. 


‘During the preparation of this paper, Mr. Platise’s website 
seems to have gone off the air. The author does not know of 
a mirror, and hopes that it will return soon 
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Gateway architecture 


To provide an interface between the system 
administrators and the instrumentation network, some 
sort of gateway system needs to be defined. As men- 
tioned earlier in this paper, the original design called 
for the use of an HTTP server, probably Apache, to 
provide this interface. The construction of simple CGI 
programs to interface to the back end network, or 
even, perhaps, an Apache module (mod_ instrumentation?) 
would be a very straightforward way to accomplish 
this. Another possibility, of course, is to use an SNMP 
agent to fill this role. A collaborative project to define 
a standard for data center instrumentation systems 
could, as part of this effort, define an SNMP MIB for 
the various sorts of interfaces which one would expect 
to find on the monitoring probes. This would of course 
provide for excellent integration between the instru- 
mentation network and existing network and system 
management tools. 


Conclusions 


While it 1s difficult, at this point, to conclude that 
the results have been worth the effort, the promise of 
what it should be able to do once complete, and the 
progress that has been made, have been very encour- 
aging. 

Despite the need to make a substantial invest- 
ment in specialized equipment and software for this 
project, these costs remain modest compared to over- 
all hardware costs for a network such as this. For 
example, a basic, Sun Enterprise 420R with a single 
processor and 1GB of memory has a list price of about 
$25,000, whereas a comparable machine built from an 
UltraAXmp motherboard will cost about $13,000; the 
marginal costs of upgrading these systems will con- 
tinue to favor the UltraAXmp, since, as we are acting 
as an OEM, the additional parts are normally pur- 
chased from distributers rather than VARs. Even with 
the discounts that can be expected for the 420R, the 
savings here can easily reach $10,000 to $20,000 or 
more per system. The situation for X86 servers and 
RAID subsystems is similar. Extended over dozens of 
RAID-equipped servers, the total advantage, in terms 
of acquisition costs, of the in-house integration 
approach is clearly on the order of hundreds of thou- 
sands of dollars. By contrast, the total investment in 
development hardware and software has been on the 
order of tens of thousands of dollars. Additionally, 
when the design is complete, the per-system imple- 
mentation costs should be well less than $500. 
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ABSTRACT 


Demands for high availability are increasing almost as fast as storage and performance 
requirements, posing seemingly impossible challenges for system administrators. VERITAS 
provides a variety of tools aimed at overcoming these obstacles, but they’re not often used 


effectively. 


We’ll show you some tools and techniques that work with VERITAS products to help you 
evaluate whether your systems are already overloaded, and what you can do to stall for time. We’ ll 
describe some environments particularly prone to storage-scaling issues and suggest changes you 
should make before crisis strikes. Finally, we'll provide some suggestions for decreasing the 
consequences of failures not preventable through fault tolerance. 


How To Tell If You’re Already Screwed, And Some 
Quick Fixes 


Gather Layout Information 


In order to make intelligent changes to improve 
your storage environment, you must collect detailed 
performance and configuration information. Most of 
the tools needed to collect this information are proba- 
bly already installed on your systems — you just need 
to use them. We’ ll also mention a few other tools and 
scripts that we and others have created to collect and 
present this data. Details on how to obtain these third- 
party tools are included in the appendix. 


The most important initial steps with VERITAS 
Volume Manager (VxVM) installations are obtaining 
information about and understanding your disk layout. 
The provided utilities do a poor job of helping you 
visualize your overall layout, so it’s essential that you 
generate your own diagram, even if it’s only on paper. 
It’s important that you include the layout of the under- 
lying storage if you’re using some type of hardware 
RAID device. 


The vxprint utility is usually your best source for 
storage layout information, although it takes time to 
understand what it means. You should also consider 
using a tool like “‘save-vxlayout”’ to regularly get a 
copy of this off of the local system for disaster recov- 
ery purposes. If you’re using EMC storage arrays, you 
might also consider using emcprint, a script which adds 
physical disk information to the vxprint output. 


Collect Performance Statistics 


It’s also vital to understand your storage perfor- 
mance characteristics before making § significant 
changes. Again, you ideally want to include perfor- 
mance statistics from underlying storage. All of you 
should be familiar with the standard Solaris disk per- 
formance utility, iostat. Most of you have probably 
also seen vxstat, which is included with VERITAS 
Volume Manager and allows you to get similar 
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performance numbers about logical objects managed 
by Volume Manager. VERITAS has other performance 
analysis tools available internally that you may be able 
to convince them to share, including vxfssar, which 
provides extensive VERITAS File System (VxFS) per- 
formance statistics. 


However, when you’re using hardware RAID, 
these commands don’t provide any insight into how 
the underlying components are behaving. For instance, 
you really need to know when cache hit rates start to 
fall off, particularly with write cache. With EMC 
Symmetrix arrays, you need to install the additional 
package called symcli. With Clariion products, use 
navicli getcache. You may need to run navicli setstats -on 
if you see that statistics are not enabled. 


Presenting these statistics meaningfully is more 
challenging, but a couple of tools make this process 
easier. Cricket is a good way of doing trends-based 
monitoring of more than just network equipment. 
We’ve found it useful for tracking disk performance, 
filesystem usage, network utilization, etc. By allowing 
you to create views of all of these statistics together, 
Cricket makes it much easier to see when you are 
starting to hit a bottleneck in one of them. However, 
when all you want to do is collect some quick perfor- 
mance statistics from Volume Manager, you might 
give vxstat2gnuplot a try. 


You should also take a quick look at your backup 
and restore times. In growing environments, it’s criti- 
cal to know how long restores will take and whether 
this 1s acceptable to your management. We’ve found 
that in certain environments, incremental restores, for 
Instance, can take many times longer than full 
restores. 


Looking At What You’ve Assembled 


Trends 


Detailed graphs of system performance and uti- 
lization can be crucial for several purposes, with 
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capacity planning probably being the most obvious 
one. Assuming linear growth, you can easily deter- 
mine when you will need to buy disks, add servers, or 
move users to a new machine. In other words, it shows 
you roughly how long you have before you reach 
capacity. You can sometimes also use these graphs to 
determine what that capacity 1s, once you have an 
example of a system that is overloaded. 


These graphs can also provide an alarm system 
of sorts for problems you’ve never run across before. 
The most common examples of this are sudden 
changes in utilization, such as those shown in the 
graph of interface utilization (Figure 1). 


Peak usage times 


In order to optimize system performance and 
availability, you need to understand when your peak 


usage occurs and how extreme it 1s. These peaks often 
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occur at somewhat different times than your manage- 
ment may believe, so be sure to collect your own data 
to see for yourself, possibly using cricket or vxstat2gnu- 
plot, as mentioned above. 


In many cases, you may find that you have mul- 
tiple peaks, possibly corresponding to users in differ- 
ent time zones. With careful planning, it’s possible to 
use your knowledge of these peaks to share resources 
more effectively. In the example above (Figure 2), you 
can see how sharing storage more effectively between 
users with different usage patterns might allow you to 
more fully utilize your storage. 


Of course, sharing resources with users around 
the world can also reduce off-peak periods, which you 
may have relied on for maintenance or to improve 
backup performance. 
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Figure 1: Example of Cricket data used to understand anomalous trends. 


328 
yi 
28 
Why a Althil 
i i TTY 
* | 
208 
CAAA yk l 
g { I 1 i ! 
¢ LAF j - \ fey }) 
§ 150 "i 
7 oy 
£ vt 
£ >| 
§ | » | 
100 | \ 4 Pil 
an % 4h ve \ Ne | 
} +) . ’ 
If? } he} WU A , 
/ aN hh “aM : "uN NR 
59 vf J SN ANA 5 ‘ \ 4 y 
i ye m ervey oa { ete 
: " 
as ; | = _ fy vol ev 
CSR oda thet , 
M fnghe en YEA? Ay Kf et er A in oa 


4368 
6:e0 
12:08 
4:08 
8:08 


2080/07/11 
2088787712 
2888/87/12 


2008, 07711 
ZnO 87411 





66724786 13:05 











16:68 
8:60 
8:88 

12:83 


26:06 


2060/07712 
28000787712 
2088/87713 
20080787713 
206667713 


Figure 2: Example of vxstat2gnuplot graph. 
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Hardware RAID Wrrite Cache Hit Rate = 100% 


This typically means that the write cache 1s over- 
flowing. In other words, the physical disks are not 
able to keep up with the data being written to them, 
and even an extremely large write cache will eventu- 
ally fill. Under these circumstances, write service 
times (average write times in vxstat) can quickly 
increase to over ten times their normal value. 


Adding additional cache is usually of limited 
benefit, since the writes still must be committed to 
disk, and will eventually fill it. The best approach is to 
spread busy volumes across more disks, which on 
some systems might prevent you from using the full 
capacity of each drive. Upgrading to faster drives can 
also be of some benefit. 


Disk Read/Write Times Frequently Exceed 100ms 


It’s hard to provide exact thresholds that indicate 
disk performance problems, however access times in 
excess of 100ms almost always indicate that some- 
thing is wrong. One thing to look for is a few disks 
with significantly higher access times than most others 
on a system. In some cases, access times as low as 
20ms may be a sign of trouble. 


Reducing The Impact of the More Troublesome 
Single Points of Failure 


Careful storage management and hardware selec- 
tion can eliminate most of the causes of long sustained 
outages, but OS, application, non-redundant hardware, 
and mischievous users remain as sources of occasional 
system crashes and downtime. The following fre- 
quently result from these situations and, in some 
cases, lead to each other. 


Filesystem corruption 


When filesystems are not unmounted cleanly, it 
1s required that they be checked for consistency. This 
process usually occurs very rapidly with VxFS filesys- 
tems because all pending changes have been tracked 
via intent logs, and thus, only those changes need to 
be validated. Anything that does not appear in the 
intent logs is assumed to be consistent. With VxFS 
filesystems, there are four states that the filesystem 
can be in during a check. 

1. The clean bit is set, so no check is required. 

2. The log replay is successful, so the filesystem 
is marked clean. 

3. The log replay is unsuccessful, requiring a full 
check. 

4. The log replay is successful, but corruption 1s 
later detected. 


When a full filesystem check is necessary, as in 
case three, it can be extremely time consuming. We’ve 
seen this take as long as 24 hours on a 500 gigabyte 
filesystem containing over 50 million files. The only 
way to reduce the time a full filesystem check requires 
is to create smaller filesystems. You should determine 
the maximum acceptable time your filesystems can be 
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down in the event of a full check and size them appro- 
priately. If you have multiple filesystems that require a 
full check, be sure to do them im parallel to reduce the 
duration of the outage. 


As for the fourth case, it should never occur. 
However, if it does, you will find out what a metasave 
is — hopefully you never will. 


Volume Synchronization 


Volume Manager’s dirty region logging (DRL) 
provides logging features similar to those provided by 
filesystem intent logs. DRL is intent logs on the vol- 
ume block level. It protects you against the need to 
resynchronize mirrored or RAID-5 plexes after a shut- 
down where the volume could not be marked clean. 
As with filesystem intent logs, there is still the chance 
that a full resynchronization 1s required, but it 1s possi- 
ble to decrease the time and performance impact of 
this. 


Volume resynchronization can take many times 
longer to complete on an active volume. Consider 
waiting for resynchronization to complete, or nearly 
complete before restarting services. You can see the 
progress of Volume Manager tasks, like resynchro- 
nizations, with the vxtask command in version 3x. 
With version 2.x, you can watch resynchronization 
progress with vxstat -f b that will show atomic copies, 
which are copies between plexes. 


When a mirror or a volume snapshot is being 
attached to an already mirrored volume, you can 
reduce the performance impact by setting a preferred 
plex. By tellmg Volume Manager to prefer one plex 
over another, it will perform all read operations from 
the preferred plex only. This is important because 
when a third plex is being attached to a volume, it will 
synchronize from only one of the existing plexes. If 
you force all other reads to occur from the remaining 
plex, overall performance of the volume should 
improve. You can determine which existing plex is 
being used to synchronize the new plex by using vxstat 
-f b and look at which disks are performing the atomic 
reads, 


Damaged Data Due To User or Application Error 


Even if you manage to protect against corruption 
due to system crashes, you still have to contend with 
damage caused by rogue users and applications. The 
traditional approach to protecting against this relies on 
regular backups. Unfortunately, restoring from backup 
usually means that you lose up to a day’s worth of 
changes, and these restores can often take a long time 
to complete. A couple of VERITAS technologies 
allow you to recover from this type of failure much 
more quickly: Volume Manager snapshots, VxFS 
snapshots, and VxFS checkpoints. 


A Volume Manager snapshot is essentially a 
write-only mirror of the volume that is broken off and 
becomes its own volume. The only performance 
impact on the original volume is during the initial 
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synchronization. Another disadvantage is that it 
requires aS much disk space as a full mirror of the 
original volume. Since it becomes its own volume, it 
assumes all of the characteristics of a normal volume 
including persistence across reboots. However, it is 
created with the nolog option set, so you will need to 
stop the volume and change this parameter before you 
attempt to associate a logging plex. At any time, you 
can mount this in place of the original filesystem to 
revert all changes or mount it and copy individual files 
that need to be restored. A Volume Manager snapshot 
is a great way to take a backup of your data with mini- 
mal impact to the primary filesystem. 


A VxFS snapshot is mounted by associating stor- 
age with a mounted VxFS filesystem. It reflects the 
state of the original VxFS filesystem (known as the 
snapped filesystem) at the time the snapshot was cre- 
ated. As the snapped filesystem changes, the original 
version of those changed blocks gets copied to the 
snapshot. The main advantage of a VxFS snapshot is 
that it uses less disk space than a Volume Manager 
snapshot, only requiring enough storage to accommo- 
date the blocks that change during the time the snap- 
shot is mounted. Unfortunately, it can be hard to 
accurately predict how much disk space those changes 
will require — the snapshot becomes disabled when the 
underlying storage fills up. Filesystem performance 
can be significantly impacted while the snapshot is 
mounted, since three operations are required for each 
change. Also, since a snapshot can only be associated 
with a mounted filesystem, it is not persistent across 
reboots. For these reasons, it 1s typically only used as 
a way of providing a consistent image during a 
backup, but they don’t provide a mechanism for 
reverting the entire filesystem. 


VxFS checkpoints are a sophisticated way of 
providing the Volume Manager snapshot capability of 
reverting a filesystem back to a specific point in time 
nearly instantaneously. They work in much the same 
way a VxFS snapshot does, but they are not mounted 
and they use the primary storage to store the changed 
blocks. They are useful for performing backups using 
the Block Level Incremental Backup product. Since 
you can’t mount a VxFS checkpoint, you can’t easily 
restore an individual file. 


Failed Components 


Volume manager does an excellent job of provid- 
ing details when errors occur. It is your job to make 
sure those notifications go to the right people. Vxno- 
tify is used to provide email notification of volume 
manager errors. It defaults to sending those notifica- 
tions to root on the local machine, but you can specify 
any email address to receive these notifications in the 
startup script, /etc/init.d/vxvm-recover, by changing 
the argument it passes to vxrelocd. Volume Manager 
also reports errors to syslog, providing another mecha- 
nism for failure notification. Syslog is the only 
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mechanism by which VxFS reports errors, so it 1s crit- 
ical to monitor syslog in some fashion, such as with 
Netcool. 


Full Filesystems 


Full VxFS filesystems can be even more trouble- 
some than UFS filesystems. In both cases, it is possi- 
ble to grow the filesystem online. However, VxFS 
filesystems need space for additional structural infor- 
mation during filesystem growth operations, making it 
impossible to grow a completely full filesystem. 


To prevent this situation from occurring, always 
monitor your filesystems carefully. If you aren’t com- 
pletely confident in your monitoring system, you 
should at least create a ~10mb placeholder file so that 
you can easily free up just enough space to grow the 
filesystem. Growing a filesystem by a large amount 
requires more space for structural information than 
growing it by a small amount, so it’s sometimes neces- 
sary to grow it in multiple steps. However, we’ve been 
wamed by folks at VERITAS that it’s safer to limit the 
number of growth operations. 


Failover 


Any application or service that maintains unique, 
persistent data requires some sort of recovery strategy. 
Depending on business needs, it may be acceptable to 
sustain a few unplanned 30-60 minute outages per 
year, with a chance of a multiple hour outage. If this is 
not acceptable, some sort of failover scheme is neces- 
sary. Automated failover becomes even more critical if 
you’re committed to providing such a high level of 
service availability that even occasional planned 
downtime is otherwise impossible. 


A failover strategy can be as simple as having an 
idle system connected to the same disk array. This sort 
of primitive approach requires thorough rehearsal and 
process documentation, but it can be nearly as effec- 
tive as more costly solutions. 


A step up from this is what amounts to home- 
grown automation, or partial automation. A fairly sim- 
ple script can be used to ping another machine that’s 
running a service, and if it’s not there, start that ser- 
vice locally. However, it can be tricky to accurately 
detect every possible failure mode, and the failover of 
persistent storage is hard to automate safely. 


VERITAS Cluster Server is one of the more pop- 
ular commercial solutions for automating failovers. 
This may be the easiest, most reliable solution if 
you’re running one of the applications or services for 
which they provide a bundled solution, such as Oracle 
or NFS. It automates things so your failover is consis- 
tent. It understands dependencies between services to 
ensure consistent adherence to failover policies. It 
parallelizes many activities, such as checking and 
mounting filesystems, which results in rapid startup 
times. Finally, it has sophisticated safeguards to pro- 
tect against what’s generally called split-brain 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Larson & Stansell 


syndrome. Split-brain occurs when two hosts try to 
write to the same disk, often resulting in corruption. 


Characteristics of Environments Likely to Experi- 
ence Extreme Scalability Problems and What 
You Should Be Fixing Now 


Lots of Small Files 
Metadata updates can be bottleneck 


The VERITAS File System stores all of its meta- 
data at the beginning of the volume, which includes 
things like directory information and the intent log. 
Certain types of filesystems, particularly those with a 
large number ef small files, can tend to be perfor- 
mance bound by the speed of the disk(s) on which the 
first portion of the filesystem is located. The perfor- 
mance of intent log writes is reduced because the 
filesystem is also performing other operations. In 
some cases, this can be improved considerably by 
moving the intent logs onto dedicated storage, which 
is what the QuickLog product allows you to do. It 
does add considerable complexity, particularly in clus- 
tered environments, but the performance gain is often 
worth the hassle. 


In some cases, vxstat may indicate that even after 
the use of QuickLog, the disk(s) that contain the first 
portion of the filesystem are having a hard time keep- 
ing up. Intent logs aren’t the only type of metadata 
normally stored at the beginning of a VxFS filesystem, 
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but they are the only type that can be moved else- 
where. As is true in most other cases where a volume 
is performing poorly, the only way of decreasing the 
impact of the remaining metadata is to increase the 
number of columns in the plex (stripe it across more 
disks). 


The simplest approach to increasing the number 
of columns is to attach a new plex that contains more 
columns (disks) than the original plex, then remove 
the original plex(es). If you are running VxVM 3.0 or 
above, you can also use online relayout, which can get 
you to the same state without requiring quite as much 
disk space along the way. Both of these methods can 
be accomplished with vxassist. 


Backups Are Hard, Restores Are Even Harder 


The only practical way to back up a filesystem 
with a large number of files is via the raw device, 
since backing up each individual file sequentially 
would take a very long time. However, it’s critical that 
you have a consistent version of the filesystem to 
work with for the duration of a backup. Snapshots, 
either at the volume or filesystem level, are generally 
the best approach. The FlashBackup extension to Net- 
Backup provides an easy way of automating backups 
via filesystem snapshots. It also has the advantage of 
working with both VxFS and UFS filesystems. 


Restores of millions of files also need to be done 
through the raw device, if at all possible, since they 


Figure 3; Sea disk re-layout aan 
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Figure 4: Example of mirrored volume with striped, concatenated, and logging plexes. 





Figure 5: Example of layered volume with sub-volumes. This is also known as a striped-pro volume. 
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experience similar performance problems to backups. 
Even though incremental backups are possible and rel- 
atively fast with FlashBackup, incremental restores are 
another matter. Incremental restores are done through 
the filesystem instead of through the raw device, as 
other restore utilities like vxrestore and ufsrestore do. 
This can literally take weeks to complete when you 
have tens of millions of files. If you have that many 
files, you’re probably better off only doing full back- 
ups, since they restore directly to the raw device. 
We’ ve even seen full restores with FlashBackup take 
less time than the original backup since the restore 
doesn’t require the inode map. 


High Transaction Rate Systems 
Extremely Busy Disks 


From the info provided by vxstat or other tools, 
you'll probably find that certain disks are being used 
in too many “hot” volumes, or are otherwise unable 
to keep up with the I/O activity. If there are only a few 
busy volumes on the system, it’s often possible to 
have only a portion of each physical disk in a busy 
volume and use the rest in more idle volumes. This 
allows you to increase the number of columns without 
having to add a lot of extra storage. 


However, if almost every volume is extremely 
busy, you may not be able to fully utilize your disks. 
This is often a problem on systems with large disks, 
where the number of transactions per second is more 
than these disks can possibly keep up with. When pos- 
sible, try to use small, high performance disks for your 
busiest volumes. If you’re stuck with larger drives, 
consider preallocating up to 50% of each drive for a 
scratch volume, ensuring that other admunistrators 
don’t accidentally over-commit them. 


General relayout 


The importance of carefully planning storage 
layouts before a system goes live can’t be stressed 
enough. However, even after a system has been 
deployed, it’s often possible to reorganize things. Start 
by creating a storage layout grid, similar to Figure 3. 


Even if you only have one free disk available, it 
should be possible to gradually move data until it 
looks like your ideal storage plan. Online relayout 
may be your only choice if you don’t have many free 
disks. It should be possible to make all of these 
changes without any service interruption, although the 
process of moving data can have a serious perfor- 
mance impact. 


Glossary 


Subdisk — A consecutive portion of a physical disk 
analogous to a partition. One or more subdisks 
can be grouped to form a plex. 


Column — Groups of one or more subdisks that consti- 
tute a single stripe within a plex. For example, a 
4-way striped volume would have four columns. 
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Plex — Typically, a set of subdisks that contain a com- 
plete copy of the data on a volume. Logging 
plexes are another type of plex which stores 
Dirty Region Logging data. 

Volume — A collection of one or more plexes which 
make up a logical disk. It can hold a filesystem, 
swap device, or araw device for a database. 


Sub-volume — Starting with VxVM 3.x, you have the 
ability to create layered volumes where, for 
example, you can stripe across a set of mirrored 
subvolumes. 
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Appendix: Third-party tools 


Cricket 


This tool is written and maintained by Jeff Allen. 
It can be downloaded from http: //cricket.sourceforge. 
com. 


emcprint 


The emcprint script produces It prints the direc- 
tor controller LUN for each disk that has an EMC 
name. This assumes you have run the createMatrix 
script (depends on symdey, included with EMC’s sym- 
cli tools, a separately licensed product). This and the 
following three utilities are available from 
http://www.vxideas.org. 


save-vxlayout 


This tools preserves critical Volume Manager 
layout information and gives you a fighting chance of 
recovery if your configuration database gets corrupted. 
This sort of corruption can be caused by a split-brain 
scenario, where two systems try to update a disk group 
at the same time. It can either periodically write this 
backup info locally (to a file on a non-shared disk 
group) or email it off of the machine. We’ve also 
found it very useful for remote troubleshooting to use 
this to collect all of this info on a central management 
machine. 


vxstat2gnuplot 


This tool generates graphs of performance statis- 
tics of various VxVM-managed objects. This also 
depends on running rotatevxstat (which 1s also useful 
for determining bandwidth requirements for SRVM). 
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drive-status 


This 1s a simple script which, when run from 
cron, notifies you of downed tape drives in Net- 
Backup. This seems like something that NetBackup 
should provide natively, but at least as of version 3.2, 
it missed this. In large NetBackup environments, 
downed tape drives are often the leading cause of 
failed backups. 


Author Information 


Karl Larson attended Harvey Mudd College, 
majoring in Biology. He accepted a job as a Systems 
Engineer for WebTV Networks in 1996, where he 
helped support their production environment and scale 
it to support nearly a million users. He left to work at 
GNAC starting in 1999, where he was a Systems 
Architect, among other roles, helping design highly 
available customer environments. He has _ recently 
started work at Tellme Networks as a Senior Systems 
Engineer. 


Todd Stansell attended UC Davis, majoring in 
Computer Science Engineering. He has been an 
employee of GNAC since early 1998, supporting cus- 
tomers such as WebTV, Exelixis, and Redherring. At 
GNAC, both Todd and Karl have worked with VERI- 
TAS Consulting to support some of their most critical 
customers. 


66 


Larson & Stansell 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


User-Centric Account Management and 
Heterogeneous Password Changing 


Doug Hughes — Aubum University 
ABSTRACT 


There are a plethora of password changing programs available for users and systems 
administrators to use. Most of the existing password changers, however, fall short in some way. 
Either they are still text based, requiring users to use unfamiliar tools in unfamiliar environments, 
or they are part of an all encompassing framework — designed to completely supplant the entire 
existing account creation, maintenance and distribution process. Our program is designed, first, to 
be usable without training or human support. Second, it is designed to use existing distribution 
databases such as NIS and LDAP. We integrate many of the features of prior password changers 
such as the Cracklib library, character classes, and rules for password selection. We try to provide 
easy extensibility both in terms of database support and by providing other user-focussed 
programs that use the same authentication framework. 


Introduction 


Auburn University College of Engineering has 
two primary platforms for supporting a broad clien- 
tele. These are Windows boxes which use SMB [1] for 
file service, and UNIX boxes which use NFS for file 
service. In order to use any network services, a user 
must be authenticated. For the UNIX side, this means 
NIS, NIS+, or LDAP. 


To provide authentication to NT boxes we use 
Samba [2]. To do this semi-securely [3] requires a sep- 
arate encrypted password database. In their respective, 
encrypted formats, the UNIX password and _ the 
Microsoft password are incompatible, irreversible one 
way hashes; there is no method to generate one from 
the other. This means neither system’s builtin pass- 
word changing mechanisms can be used. This irre- 
versibility problem can be applied by extension to 
other one way password formats such as BSD44, 
Apple formats (e.g., CAP [4]) and several Linux for- 
mats. 


Early in 1999 we decided that we should replace 
the existing — functional but archaic — text-based pass- 
word changer with a Web-based mechanism. The old 
system required users to login to a UNIX machine to 
change their passwords. Many of our new users had 
no experience with applications like telnet, ssh, or 
even UNIX. Text based solutions requiring login to 
an unfamiliar environment were difficult to support. A 
Web design allowed greater platform availability and 
improved usability over our previous system. We also 
had an opportunity to revise the entire password 
changing system and integrate other services such as 
email forwarding. 


We wrote our code primarily in PHP [5]. Some 
functions required a PHP loadable module written in 
C. One helper program was written in Perl. 


Prior Art 


A lot of work has been put into account manage- 
ment systems over the years. Many USENIX, LISA, 
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and even ,/ogin: articles have been written concerning 
these works. Nearly all of them have a systems admin- 
istrator focus, which is natural when one considers 
that systems administrators are always trying to make 
their jobs easier. However, the works tend to focus on 
systems administrator specific tasks such as creating, 
adding, deleting, and ongoing maintenance of 
accounts. When usability was addressed, it was usu- 
ally most evident from a systems administrator’s point 
of view: make less work; provide more scalability. 


Auburn University College of Engineering (here- 
after referred to as AUCOE) already had account 
maintenance tools in place with no obvious need for 
wholesale replacement. We required an intuitive user 
interface — to allow users to be able to do simple tasks 
on their accounts without any sort of training — and 
minimal support. 


The main differences between the AUCOE 
framework and prior implementations are: 

1. It integrates prior work and tools such as 
Cracklib [6], character classes, etcetera. (some, 
but not all, of the others also do this.) 

2. It provides easy extensibility to other formats 
such as LDAP and custom databases. 

3. It is much simpler in scope than the others. 
Some of them are enormous and handle every- 
thing from account creation to interfacing with 
a human resources database. The AUCOE 
changer provides a means for users to easily 
change their passwords. 

4. Our main focus is user usability vs. administra- 
tor usability. 

5. The incremental approach builds on top of 
existing frameworks such as NIS, NIS+, 
Samba, and flat files, rather than replacing 
them. 

6. Decoupling password choosing from distribu- 
tion and propagation allows us to not have to 
worry as much about locking and conflict reso- 
lution. 
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The author was not able to gather meaningful 
data — because of age and lack of access to the original 
papers — on these password and account management 
systems: Maryland [7], ACMAINT [8], Apoilo [9]. 
References are provided at the end for those having 
hardcopy proceedings. 


Many of the features of the following command- 
line, pro-active password checkers have been inte- 
grated into our password changer: npasswd [10], 
passwd+ [11], ANLpasswd [12], Epasswd [13]. The 
password vetting routines from these programs tend to 
be tightly coupled with an existing command-line 
interface and propagation mechanism. Rather than 
extracting the proactive changing logic and rules from 
any of these programs, it was easier for us to use 
Cracklib and construct the various password rules and 
classes in native PHP. 


Information about current availability of and 
updates to other user account management systems 
was difficult to collect. Some of the systems listed 
may have had updates more recent than the author was 
aware. Some of them were designed to manage access 
control where certain users are only authorized to use 
certain machines, or the home directory may differ 
based upon the machine, All of them had, as a primary 
focus, the goals of modernizing and automating the 
maintenance of user accounts at a site; password 
changing, password synchronization, and usability, if 
mentioned at all, were typically secondary. The most 
currently relevant systems are compared in Table | 
based upon the following criteria: 
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e Design philosophy 

e Custom database vs. existing database (e.g., 
NIS, LDAP) 

e Distribution and synchronization methods for 
accounts and passwords 

e Extensibility — perceived degree of difficulty of 
adding a new output format, heterogeneous 
machines, configuration intricacies, etc. 

e User interface focus 

e Language(s) written using and/or configured 
with 

e Release status (may be out of date) 

e Other characteristics that may be of interest 


Genesis 


The framework used by the AUCOE password 
changing system came into being ad hoc. The project 
that originated the framework provided a means by 
which users could forward their email. The previous 
forwarding program was text-based and required users 
to login to a UNIX machine to run it. The number of 
users wishing to forward their email was growing. The 
time spent supporting these non-UNIX users was 
growing proportionately. 


We desired to avoid CGI and its inherent security 
difficulties and call-out overhead. After a short period 
of investigation, comparing the tradeoffs of mod_perl 
[19] versus learning PHP, a new language, the author 
chose PHP. PHP combined C and Perl syntax without 
the special variable cruft. PHP had support for persis- 
tent file handles. Finally, PHP appeared to be easier 
for non-experts to lear and use quickly. 


Design Database Distrib./ | eee 


Shuse [14] sync accts across 
mult mach’s with 
central database 

Agus [15] acct. over multi 
archs using CCSO 
[13] 


custom but open 


NAMS [16] client/server dae- 
mon with modules | (ASCII key & da- 
replaces NIS ta) 


ease hiring process 
and account cre- 
ation 


Accountworks [17] 


Ganymede [18] provide central DB 
push to NIS, 


LDAP, etc. 


Aubum COE Provide usability use existing (NIS, 
atop existing dist. etc.) 
mechanisms 


custom, central 


custom (fixed?) 


Sybase (other..) 


Central, RAM, OO 





NFS, NIS, FTP, 
sockets 


uses expect 


Kerberos, VMS, 
Unix, etc. 


fixed DB but godo 
for new accounts 


TCP/IP socket 
(client/server) 


modular design 


central client/serv- | probably difficult 

er but no passwd 

sync 

NIS, NIS+, LDAP, | properly changing 

etc schemas not for 
timid 

use existing (NIS, Use PHP, module, 

etc) or other (e.g., 
TCP/IP) 


Table 1: Survey of similar systems. 
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In keeping with the tenets of user-interface 
design, the password change form was designed to be 
simple but usable. User feedback was incorporated to 
make the form what it 1s today (Figure 1). The results 
were positive both in terms of user satisfaction and 
reduced support by administrative staff. We were 
encouraged to try again: with passwords. 


Design Goals 


Our system for changing passwords was 
designed to meet certain goals: 

e It must have universal accessibility. 

e It must be secure. 

e It must be easy for novices to use. 

e It must not rely upon extensive online help. 

e It should have access to online help for users 
having trouble. 

e It must not inhibit expert users. 

e It must be scalable. 

e It must provide clear and meaningful, jargon- 
free responses. 

e It should build upon existing distribution and 
synchronization mechanisms: NIS, NIS+, 
Samba, LDAP. 

e It should be easy to extend as other distribution 
and synchronization mechanisms become avail- 
able. 

e It should borrow techniques from prior pass- 
word changing implementations. 

We now provide finer detail on these goals. Why are 
they here? How are they achieved? 


Nae | User Construct/ | i 
| Interface _ Config 

Tcl/Expect with a 

little bit of C 





Shuse [14] text based 


~ unknown 


text based (cus- 
tomized npasswd) 


NAMS [16] 


Accountworks [17] | Web based 









etc. 





Ganymede [18] Java Applets, 


count on training 





Aubum COE Web forms 





90% Perl, 10% C 


C daemons, ASCII | prev. 
DB config 


DB driven + Perl, 
sybperl, Notes, sh, 


140K-+ java lines, 
web-based config 


PHP, some C 
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It Must Have Universal Accessibility 


In other words, use the Web. It is time consum- 
ing to develop clients for various architectures. The 
flexibility and speed of dedicated, platform-dependent 
password clients was not indicated. 


It Must Be Secure 


Because users are. providing information that 
gives access to their accounts, and because secure, 
local access cannot be assumed (though it could be 
enforced), it is imperative that no information be 
transmitted over the network in the clear. There are 
multiple paths of transmission involved. See the Secu- 
rity section for more details. 


The Web server must verify the username and 
old password for authenticity. Our implementation 
provides multiple ways to do this. 

1. Make the Web server a NIS slave. 

PHP can directly access DBM files, so verify- 

ing passwords by giving the Web user account 

read-access to the encrypted password database 
is possible. (Before anybody gets apoplectic, 
please continue reading.) If your Web server is 
single-purpose, contains no extra CGI, no other 
user scripts, and no unknown functionality, this 
is relatively safe. 

2. Use the provided pwcheck daemon. 

The pwcheck daemon uses a secure algorithm 

to verify the username and password. If neither 

is correct, a non-specific negative result 1s 
retumed: users are informed that either their 
username or password is incorrect. 










Release Other 
Status Features 
Only Sheridan 








College 













account clusters, 

fine-grained ma- 
chine access con- 

| trol 


N/A at publication | 












account clusters 





ftp.cs.jmu.edu 









designed for non- 
techies; huge scope 


| not available 












DB limits, good 
access control 


www.arlut.utexas. 
edu/gash2 





easy to add new 
features, e.g., for- 


www.eng. aubum. 
edu/ doug 


ward mail 


Table 1b: Survey of similar systems, cont. 
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3. Use another authentication mechanism. 
While not provided, it would be fairly easy to 
interface with LDAP, NIS+, or another authen- 
tication database to verify a user’s identify. 
One of the difficulties of this approach 1s 
choosing between the lesser of two evils. Do 
you wish to give your Web user root access to 
the entire authentication database or do you 
wish to provide a setuid root helper program 
that becomes the user prior to checking the 
encrypted password? This is the primary reason 
why the pwceheck mechanism 1s provided. It is 
discussed in more detail in the Security sec- 
tion. 
It Must: 
Be Easy For Novices To Use 
Rely Upon Extensive Online Help, and 
Provide Access To Online Help For Users 
Having Trouble 


These three goals are complementary, and 
might even be merged into one. The distinctions 


File Edit View Go Communicate 


5 a Boohrmike A Le Location: https: //imapss]/passvd. php? 


i he akaeies a ¢a 3 & 


Pret Seorky Sup 


+ 
| 
rH 
Cs) 
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among ease of use, having online help, and relying 
upon online help, though subtle, are important. The 
interface must be intuitive. One of the well-known 
facts of user-interface design is that users rarely, if 
ever, read any online help before attempting to do 
something. They usually dive in oblivious to the cir- 
cling sharks, The interface must be intuitive. 


If a user cannot sit down and use a password 
changing program without training or referring to a 
manual, the program should be re-evaluated. We 
have gone through several iterations of user feed- 
back and modification. There is, however, a differ- 
ence between using the program without requiring a 
manual and referring to supplemental help should 
the password choice be insufficient. We avoid the 
requirement to read online help before-hand by pro- 
viding abbreviated rules for choosing a password 
right at the top of the Web form. The rules (Figure 2) 
are specific enough to be easily understood, but 
small enough to fit entirely on the form and jn the 
browser window. 


Te z: Help 
C5" Wrats Felated 





om 


Your password was rejected because it is based on a dictionary 
word. Please choose a better one or follow this link fot advice. 


Change CoE Password Form 





e Passwords must have six(6) to eight(8) characters 
= mustcontain at least one(1) uppercase letter 
= mustcontain at least one({1) lowercase letter 


= must contain at leaSt one(1) other character 
egOl23456789I@#SB*&-_ +-Vin<?/-*""' 
@ Your password cannot be a word found in any dicdonary! (English, 


Foreign, or other) 


For hints and clues about Soooens a good password that won'tkeep getting, 


ore i 


APT Neer eee see OURS e mmmmespeeettercemess 1s 


Username: = Houg Please enter your CoE username. 


Old Password: [Rannaans RARAAKA = Please enter your old Cok Password. 


New Password: i [A ApakeE Please enter your new CoE Password. 


New Password(again): Ih AR ARK KA Plense enter you new password again. 
| Change Password ‘Reset| 





Figure 1: Sample user interface. 


70 2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Hughes 


After each rejection of a password based upon 
the given rules, the user is given the opportunity to 
follow a link which gives suggestions for and exam- 
ples of choosing a password. Failure messages are 
communicated in bold red letters in a larger font and 
written at the top of the page as the form is redrawn, 
giving clear visual feedback about problems. 


It Must Not Inhibit Experts 


By providing a short set of rules, but not over- 
whelming the user with extensive directions, the 
interface 1s usable by experts. 


It Must Be Scalable 


Most of the scalability (and locking) issues are 
pushed back to the distribution mechanism and thus 
avoided. The interface must still, however, be able to 
handle the case where a number of users connect 
simultaneously. The worst case scenario can proba- 
bly be attributed to university environments when 
new accounts are generated at the beginning of each 
term. Even in this scenario, however, it is extremely 
unlikely for more than a few users to change their 
passwords at a given time. Many of our users opt to 
keep the random FIPS-181 [20] style passwords that 
they are issued. Even if there was a Freshman com- 
puter lab where the instructor dictated that all stu- 
dents must change their passwords, changes would 
still not be simultaneous. People read, type, and 
think about their passwords at different rates. The 


https request 






check pass 


use NIS auth status use pwcheck 





NIS slave files 





status 


HS 


auth status 
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worst case would probably not be more than 5 to 10 
simultaneous password changes, which 1s easily 
achieved even with a modest PC as the Web server. 
In practice, even in its 18 months of existence, it has 
been unusual to have more than one person trying to 
change a password in any given five minute interval. 


Locking issues, similarly, are not a great con- 
cern. Most of the methods to change passwords in 
the existing back-end databases are already serial- 
ized. NIS, NIS+, and LDAP already have their own 
locking mechanisms. Even when we update our 
Samba password file with the encrypted MD5 
hashes, we use a simple, Perl, single-threaded dae- 
mon to handle network requests from the Web server 
and update the flat Samba password file. (See the 
Extensibility section). 


It Must Provide Clear and Meaningful, Jargon- 
free Feedback 


User feedback issues have been partially previ- 
ously addressed. At the risk of appearing to climb 
onto a soapbox, this is an area that 1s ignored far too 
often by systems administrators developing user 
interfaces for users. Users do not care — and should 
not be burdened ~ with messages about errors in 
some anonymous line of computer code. We systems 
administrators, as a community, are particularly 
guilty of these transgressions. The AUCOE program 
gives messages in plain English such as ‘New 


https response 






| 


ey auth 


---- 


update — auth dbs 


L update 








success or failure 


—update 





update 





smbdaemon } 





samba pw file 


Figure 2: Password verification and update flowchart. 
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passwords do not match. Please retype them.’ , or 
‘Passwords require at least six characters’, or “The 
new password you entercd its too similar to your full 
name or username.’ Upon successful completion, 
the program explicitly tells the user: ‘Your password 
has been changed successfully and will be ready for 
you to use anywhere in engineering in the next 10 
minutes.’, giving them explicit expectations and 
scope. 
It Should Build Upon Existing Distribution And 

Synchronization Mechanisms 

One of the drawbacks — or features, depending 
upon your pomt of view — with some of the other 
systems is that they require wholesale replacement 
of existing mechanisms. This means modifying 
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every machine, or replacing the entire account gen- 
eration mechanism, or tailoring lots of configuration 
files. Sites starting with nothing, sites having special 
access requirements, or sites mired in complicated 
legacy scripts may have no qualms about replacing 
an existing system. We provide an incremental 
approach to sites that, like us, already have estab- 
lished account generation and maintenance mecha- 
nisms. It is not always possible to build incremen- 
tally, but in our case it was desirable. 


It Should Be Easy To Extend as Other Distribu- 
tion and Synchronization Mechanisms 
Become Available. 

PHP, like Perl, is a "kitchen sink" language. It 

contains builtin interfaces to many databases and a 


// Who do you want administrative mail (errors, etc) send to? 


SAdmin-Statt = "admin": 


// General Vars - length of passwords 
SMAXlength = 8; 
SMINDene th: =. 6; 


// Error logging (daemon|error) (consult syslog.h) 


SSYSLEVEI: = 27 


// comment this out if you don’t have NIS.. If you do this, you 
// better have LDAP (er skills to modify passwd. php3) 
// Note - this is for submitting the password change. 


Shave_NIS = 1; 
SNIS- domain: = “eng .auburn.edu":: 


// This is for checking the current password 


// For machines with direct access to shadow password file, 


i} yset-Sda rect access: — 1; and set ‘password tile Jike the following 
Wi ane 
// Spw_dbm = "/var/yp/domainname/passwd.adjunct.byname"; 


Sdirect_access = 0; 


// Where do you keep your user .forward files? 


Sforward path =—°"/forwards": 


sf} IP -youcuse. the forward-fiunction, choose. the-.location..of your plpcfile 


// helper auxiliary program 


Sphp_forward_helper = "/etc/local/php_file":; 


i} For getting page count. database: intca, Basically «nobody really 
// eaves about this but us (Auburn). 1t’s used: im paper. php3. 


Sgetpages = "/etc/local/getpages"; 
// Change these for LDAP 


// ldap_uid should be an account with create and modify privileges on 
// everything in your Samba and NIS ldap trees. 


shave ldap = ol: 

sidap.server = “ldap; your, domain”; 
Sidap ind =. “Took: 

Sidap! pw = “noorepw'; 


Sldap_smbdn = "ou=people,dc=eng,dc=auburn,dc=edu"; 
Sldap nisdn = "ou=pecple;de=en¢e .dc=auburn.dc=edu": 


// Samba tree 
// NIS/UNIX tree 


// For socket based manual updates to an smbpasswd file, this is 
// the host. Uncomment it here and change it as appropriate 


// SSMBHOST = “smbhostname"; 


Figure 3: PHP configuration file. 
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rich set of string processing and system interface 
functions. It also has a well-documented and easy to 
use C API for extension. The Cracklib module is 
written as a loadable extension, as are methods for 
generating Samba hashes, and calling NIS for pass- 
word update. TCP/IP sockets and cryptographic 
functions are also builtin providing yet another 
means for extension: client/server communication. 


Architecture and Implementation 


The code, where possible, was written in PHP 
for Apache [21]. A dynamically loadable PHP mod- 
ule was constructed as an enhancement. We also 
created a Perl helper daemon on the Samba PDC to 
accept network connections from the secure Web 
server and update the Samba password file. 


Figure 2 shows a high-level flowchart of a 
password change transaction. The ‘auth method’ box 
encapsulates all of the functionality for checking to 
see that the usemame and old password are valid, as 
well as running the rule checks and Cracklib on the 
new password. Upon successful validation, the 
respective authentication databases are updated and 
status is returned to the user. The dashed "valid auth" 
line is conditional upon success of the authentication 
method. The dashed "update" line to the other box 
represents generic extensibility. 


Configuration is currently managed through a 
PHP include file: globals.php3. Figure 3 shows a 
sample, commented globals.php3. This file lets you 
define and configure your authentication services so 
that when a user changes a password, the appropriate 
databases are updated. 
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e Between the Web browser and the Web 
server. 

e Between the Web server and the pwcheckd 
daemon (if used). 

e Between the Web server and the authentica- 
tion database (during updates). 

e In ancillary programs. 


On The Web Server Machine 


Changing passwords has broad security impli- 
cations. We strongly recommend that you dedicate a 
machine to these ‘user services’. Do not install this 
software on your general Web server. Do not give 
users interactive accounts on the machine itself. 
Users given direct access to the machine may have 
access to files that they should not. 


Browser to Server 


As discussed in the goals section, the transac- 
tions between the client browser and the Web server 
must be encrypted using HTTPS, preferably using 
128 bit clients. You should force your Web server to 
only accept high grade security connections. 


If you decide to setup the Web server as a NIS 
client, you will authenticate users by directly com- 
paring their encrypted passwords in the shadow 
password file with their passwords passed from the 
browser. In this case, you do not need to worry about 
passing the user’s password over the network 
(because the files are local — except of course during 
periodic NIS synchronization updates). You do, 
however, need to be doubly certain that you do not 
give any users shell access to the machine. Addition- 
ally, you should setup NIS to use shadow passwords. 


Pwcheckd 


The provided PHP loadable module is written 
in C. There are four main functions in this module. 
The checkpass function is used to verify the user- 
name and password as accepted from the user's 


Pwcheckd has been designed to allow the Web 
server to verify the password of a user without passing 
the clear text or the encrypted version of the password 
over the network. The authentication transactions are 


browser. It calls pweheckd on the server as detailed 
in the Security section. The crack function calls the 
Cracklib library with the user’s desired new pass- 
word. The passwd function calls the RPC yppasswd 
function to change the user’s password in the NIS 
password map; the NIS master is set at compile time 
in the Makefile. Lastly, the smbpasswd function 
takes the user’s plain text password, creates the NT 
hash pair, and returns the ASCII representation to 
PHP for transmission to the s»bdaemon program. 
Transmission is accomplished with standard sockets. 


Smbdaemon is a simple Perl program running 
on the Samba PDC machine. It is a 130 line Perl 
program at this writing. It listens on a well known 
port for updates, and then writes them to the private 
Samba smbpasswd file. 


Security 


There are five major areas where security 
needs to be addresses: 
e On the Web server machine. 
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summarized in the following list. 

1. The Web server calls a function to verify the 
usemame and password which in turn calls 
pwelient in the loadable module. 

2. Pwelient connects to pwelreckd on the desig- 
nated port of a known server and asks for the 
UNIX password salt [22] for a given username. 

3. Pwcheckd checks tcp_wrappers for valid client 
access. 

4. Pweheckd sends the salt back to pwelient. 

. Pwelient uses the salt and UNIX password 
algorithm to encrypt the uscr’s supplied pass- 
word. 

6. Pwclient makes an MDS hash of the user’s 
encrypted password and current time and sends 
both hash and time to pwcheckd. 

7. Pwcheckd receives the hash and time’ from the 
user, and compares pwclient’s time with the 


Nn 


"You should use a time synchronization program like 
NTP [23] between the Web server and the machine run- 
ning pwcheckd to keep the clocks synchronized. 
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server’s time. If the time 1s not within plus or 
minus 30 seconds, the attempt ts logged as a 
possible replay attack and the user is not 
authenticated. 

8. Pwcheckd makes an MDS hash of the user’s 
actual password and time supplied by pwcliens 
and compares with the hash from pwelient. 
Success or failure is retuned to pwelien. 


Server to Database Updates 


Server to database update security is going to be 
dependent upon the authentication databases used. For 
LDAP, the current implementation uses the LDAP 
root password to open the database and update the 
encrypted portions of the user’s LDAP entry. This 
password ts currently stored in the globals.php3 file. 
Since globals.php3 contains only PHP variable assign- 
ments, as long as you do not have your Web server 
setup to allow people to fetch PHP3 files, it will be 
safe. To bring home an earlier point: do not give users 
shell access on this machine. 


For NIS, PHP calls the RPC yppasswd function 
(in the PHP loadable module) which exposes the 
user's old password in clear text while the new pass- 
word is transmitted in encrypted format. Because the 
update is fast, the risk is small and mitigated in other 
ways. (You do have your Web server and NIS server 
on the same secured, Switched network, right?) 


You may be wondering why the pwcheckd mech- 
anism appears to be so much more stringent and 
secure than the database updates. The underlying 
authentication databases have a considerable influence 
on the update mechanisms available. The author hopes 
that these mechanisms can be improved in the future. 
For now, keep your Web server and your master 
databases on the same, secure network — preferably 
locked in a room or closet with no user accessible 
jacks or VLANs. Even if you do not follow this 
advice, the exposure of an encrypted password to pry- 
ing eyes is typically* less of a concem than exposure 
of the plain-text original. 


Ancillary Programs 


For the contributed Samba password update pro- 
gram, the NT hashed password pair Is sent via a socket 
to a server program running on the Samba PDC which 
replaces any existing Samba entry with the new one. 
This 1s a generic mechanism that could easily be 
extended or replaced. As configured, smbdaemon only 
accepts connections from the Web server. No plain- 
text password is transmitted, but the NT hashes are 
plain-text equivalent in that, if they are stolen, they 
will give access to the server as that user (a well- 
known Microsoft problem). 


There is another ancillary program, php_file, not 
directly related to the password changing functional- 
ity. If you do not wish to use the mail forwarding 


| 2See following paragraph. 
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functionality, remove formail.php3 and the php_file 
program. Since Web servers are typically run either as 
the user nobody, or as a special account such as www 
or www-data, the helper program must be setuid to be 
able to edit the user’s .forward file. While the suevec 
functionality of Apache would appear to be suited for 
this task, 1t has a number of shortcomings that make it 
less ideal. 

¢ Suexec is not installed by default and requires 
special re-compilation of the Web server in 
many distributions, This, in turn, requires addi- 
tional configuration, care and knowledge. Our 
set of tools is intended to be easier to nun out of 
the box. 
The program to be executed by suexec must be 
resident in the WWW space and owned by the 
effective user doing the executing. Instituting 
this would require that all users have a copy of 
php_file in their Web directory space. This 
could be automated, of course, but would 
necessitate yet another process during account 
creation and deletion, as well as a small waste 
of space. This residency requirement could 
potentially involve thousands of new directo- 
ries in the WWW space as well; since we rec- 
ommend a dedicated Web server, a new direc- 
tory would need to be created for every user 
and the suexec-able php_file must be copied 
into that directory. 
Altering the suexec code is potentially haz- 
ardous. There are many warnings about doing 
so. Though constructing new setuid code has its 
own perils, the author opted for the perceived 
simplicity of this approach. The alternative was 
altering a complex program covered with vir- 
tual no trespassing signs and barbed wire. 


Consequently, php file is setuid root. The user- 
name and password are given to php _file to verify. In 
its current implementation it lacks the authentication 
flexibility of the password changer. Instead, it uses the 
system to fetch the user’s actual password given the 
supplied username. It then encrypts the supplied pass- 
word and compares it with the actual password. If they 
do not match, the user is informed. If the actual pass- 
word and supplied, encrypted password match, 
php_file becomes the user (setuid). During the process 
of writing a new .forward file or removing an old one, 
it makes sure to avoid symbolic link replacement 
attacks’ by calling atomic functions. The user is 
expected to own the directory where the .forward file 
is located and the .forward file itself. If any of these 
conditions is not satisfied, the user is informed of an 
error, and a sys/og(3) is sent about a possible attack, 


Extensibility 


By choosing PHP and its broad base of support 
databases and functions, much of the extensibility of 


3When a cracker exploits a race condition among system 
services in combination with symbolic links to cause unin- 
tended effects. 
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this project is builtin. Using network sockets for 
client-server interprocess communication is trivial. 
Accessing a NIS encrypted password database is as 
simple as granting permissions to the DBM file and 
opening it. Likewise, LDAP and encryption functions 
are present. Even so, we needed to construct a load- 
able module to provide a few supplementary func- 
tions. 


Luckily, PHP makes it particularly easy to 
extend itself via dynamically loadable shared object 
libraries. The AUCOE supplied module contains some 
RPC clients, a Samba hash generator, and a wrapper 
for Cracklib. 


The Perl Samba password changing daemon is 
provided as a generic model for quick and dirty exten- 
sibility. It receives connections from the network 
using a simple socket(2) accept(3) loop, verifies the 
connection is from the Web server, reads the record 
from the network connection, and writes it to the pri- 
vate Samba smbpasswd file. It also does some mem- 
ory caching optimizations. (see the code). 


Individuals wishing to add further functions in 
PHP should take a look at the Idappw.php3 file and 
see how all of the LDAP functionality is encapsulated 
into a single object. Multiple LDAP server connec- 
tions can be instantiated independently. This is the 
model that future extensions should use, and that the 
rest of the code will, eventually, be re-written to use. 


Limitations and Future Directions 


The code has not undergone significant outside 
testing. It should have an AutoConf(1) configuration 
for choosing options and a better installation proce- 
dure than copying files. More of the code should be 
converted to a class/object interface like the LDAP 
framework. The password verification mechanisms 
could be made more generic by allowing more choices 
like PAM(3) and nsswitch.conf(4). 


We constructed a GUI front-end and middle- 
ware between the login and the various independent 
functions. This allows us to offer an easy way for 
users to login once at the beginning and automatically 
get access to the various functions (password chang- 
ing, forwarding mail, access printer accounting mn for- 
mation, generating one time passwords, etc.) How- 
ever, it is currently not as comely as it might be and is 
fairly site specific. 


We have also thought about integrating the rule- 
based configuration language of something like 
npasswd or passwd+, but given the extreme thorough- 
ness of Cracklib, it may add marginal benefit. 
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Availability 


The code is currently available via anonymous 
ftp from ftp.eng.auburm.edu in pub/doug/A UCOEpw. 
tar.gz or available from the author’s tools page at 
http://www.eng.auburn.edu/“doug/second.htm| 


It 1s currently in production beta state — it works, 
but needs lots of configuration via globals.php3. The 
code is known to work on Solaris2.6 and above and on 
Linux, specifically Debian. 
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ABSTRACT 


Here we describe Pelendur, a system for the management of common system operation tasks. 
Specifically, Pelendur focuses on the management of user accounts and related information (such 
as groups) across platforms and even for particular software packages (like databases) that require 
user authentication. Pelendur has reduced the massive process of deleting expired accounts and 
creating new accounts between terms from a week-long operation by several part-time operators 
(with subsequent cleanup by a collaboration of instructors and staff) into a completely automated 
process that requires less than 15 minutes of staff work and completely eliminates the need for 


instructor intervention. 


Introduction 


In 1998, our department was in the midst of a 
massive migration of our computing facilities wherein 
we moved from an architecture of many HP-UX clus- 
ters to an architecture using Solaris-based function- 
specific servers with thin clients in offices and labs. 
Some of the software used in the old environment 
needed to be replaced [3]. Because of severe limita- 
tions in the functionality and correctness of the largely 
ad-hoc scripts run by operators for the creation of 
accounts, it was determined that an account manage- 
ment system capable of managing our evolving multi- 
platform environment was needed. 


Yet Another Account Management System? 


The idea of implementing a software system for 
the management of user accounts is not new; past 
years’ LISA conferences have seen many such sys- 
tems. Even if we briefly ignore the issue of availabil- 
ity, Some systems described were unsuitable because 
of extreme differences in the way that accounts are 
created and managed [1, 5] and incompatible means of 
handling account data [7, 11,9, 10, 4, 8]. 


In the end, the most compelling reason for us to 
build our own software was that a grander vision 
existed: a single data repository for our department, 
which would include such things as data needed for 
user accounts, course-specific computing require- 
ments, and access to various limited-access depart- 
ment resources. We could not find any available 
account management system that would work easily 
with any sort of database that we would construct. 


The Academic Environment 


Account management, though a fairly straight- 
forward task, 1s quite demanding in an academic envi- 
ronment. Each term, we receive course rosters from 
the university registrar. Although students who major 
in computer and information science have “‘perma- 
nent”? accounts (those that will remain until after they 
graduate or change majors), we have thousands of 
other accounts that are created specifically for the 
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duration of the term. We have one week between most 
terms, which means that during this time, we need to 
delete potentially more than 2,500 accounts and then 
create another 3,000 accounts for the next term. 


In this paper, and in our environment, we use a 
term that will be important to understand: “‘section’’. 
This is a specific “class” that meets together. This 
term is introduced because many sections of a given 
class can be scheduled for the same term and we need 
to know the most granular level of grouping available 
from the registrar. 


System Requirements 


Specific requirements for this system were iden- 
tified. In its full design, our scope is actually much 
more broad than the rather specific task of account 
management. The reason for this is that parts of 
account management (such as determining how much 
quota to associate with an account) are dependent on 
other criteria like the requirements for courses that the 
user of that account 1s scheduled to take. For example, 
a course that deals with particularly large data sets 
might have a requirement for more than the default 
amount of disk quota. 


Initial requirements focused on the actual man- 
agement of user accounts and dependencies. 
Account Management: This is the management of 

individual account profiles. Adding, editing, 

expiring, and purging them. 

Course Management Courses have particular 
requirements (such as the need for one platform 
or another and the use of a software system like 
Sybase) that need to be configured and man- 
aged. Some of these configuration options are 
simple matters of preference. Others are matters 
of policy, which the course coordinator does 
not have the authority to change. (Though we 
refer specifically to courses, there is nothing 
that prevents these entities to be managed from 
being project groups, departments, or any other 
sort of group that might have particular require- 
ments in other environments.) 


a 
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Resource Management: Anything that exists in our 
environment (such as “‘Unix machines”, ““NT 
machines’’, “Sybase database’’, “‘disk quota’’, 
‘print quota”’, and “‘color printers’”’) might also 
need to be managed. As these are neither 
accounts nor courses, but share basic proper- 
ties, we classify all of these as ‘“‘resources”’. 
The basic point here is that users who are 
administratively responsible for these resources 
manage them through Pelendur instead of hav- 
ing us perform all of the management tasks for 
them. 


Additional requirements were slated for future 
development. Some of these features are now imple- 
mented and others are still on our ‘‘to-do”’ list. 

Unix groups: As we use groups to manage sets of 
users on the Unix systems, Pelendur should be 
aware of groups and know how to use them. 
(This interface is now partially implemented; 
new groups are created, but old groups are not 
garbage collected.) 

Mailing lists: Some of our course instructors and 
students prefer to use mailing lists to stay in 
touch. We presently use both faculty-main- 
tained aliases files and Majordomo [2], but are 
now investigating the possibility of replacing 
both of these mechanisms with Mailman [12]. 

Course directories: Because some courses have 
group projects or software that is specific to the 
course, we need to be able to associate directo- 
ries with a given course. Handling of filesystem 
permissions should enforce the policy for read 
and write access established by the instructor. 

Multidomain management: Currently, everything 
that is a part of the instructional environment is 
considered ‘‘the system”’. Accounts that belong 
to individual research labs are not managed by 
Pelendur but it could be convenient for us to 
have that option available. Should we do this in 
the future, it would be nice to have the option 
of having a single Pelendur installation be able 
to manage multiple “‘systems”’, rather than hav- 
ing to make a new installation of the software 
for each domain that needs to be managed. This 
feature would also be useful to include access 
to limited-access machines, such as those that 
are set aside for long-term computationally- 
intensive jobs. 

Electronic lock systems: A relatively new addition 
to our department is an electronic lock system, 
whereby ID cards are used for access control 
instead of physical keys. Presently, this 1s man- 
aged by a standalone DOS-based system. 

Course newsgroups: Most courses in our environ- 
ment is assigned a newsgroup on our local 
news server. This is the default means of pro- 
viding an “out of class” communication chan- 
nel. Presently, we just create new groups as 
new courses are added and cancel the messages 
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in the groups at the end of each term by hand. 
Though this is not a big time-sink, we would 
like for these processes to be automated. 

Course-specific environment: Some courses have 
particular environmental needs, such as a 
course-specific $PATH setting, for example. Pelen- 
dur can provide this. 

“Any computing resource”: As we continue to move 
forward, other resources are identified and incorpo- 
rated into the system's functionality. At a very high 
level, the goal of the system is to manage the sys- 
tems’ configurations so that system administrators 
can do other things that computers can't do very 
well, like planning. 


Design and Implementation 


Pelendur is a large system, made up of several 
programs. We’ll first describe the philosophies that 
influenced the system’s design and then consider the 
programs and major library modules that these pro- 
grams use. . 


Data-Driven 


The entire system sits atop a Sybase relational 
database. Rather than creating code that would depend 
upon very specific data, working with entities that 
make sense for our environment, we opted to put as 
much of the system in data as possible and to make 
those data be as generic and flexible as possible. Thus, 
rather than dealing with courses, sections, and instruc- 
tors, other users of Pelendur will be able to work with 
teams, departments, and project coordinators. Each 
deployment of Pelendur will define its own terms and 
the relationships among them that make sense. 


We believe it important to emphasize that this 
makes the integrity of the database especially critical. 
In such a highly dynamic system, we’re not dealing 
with simple cases of the Wrong Thing failing to 
achieve the desired result. Data that have been com- 
promised by a moderately clever attacker can be used 
to attack the system itself, creating accounts for 
attackers, granting them privileges to the entire sys- 
tem, and possibly even running commands with supe- 
ruser access. 


The schema is represented in Figure 1. Here we 
describe each of these tables in some detail. 

People contains information about a person. When a 
person is added to the table they are assigned a 
database identifier and a user login. Those two 
values will be used to tie a person to their 
resources. 

Account is a table that contains information about 
individual accounts. 

Classification contains information on thingies in 
the system. A thing can be anything with the 
exception of a person or an individual account. 
Most classifications define a resource or a 
membership group. There are also templates 
and defaults in Classification that are used to 
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create resources and membership groups or 
define certain values in the system (e.g., what 
the current quarter is). Each classification 1s 
assigned a unique number (SID) when it is 
added to the database. 

MembershipIn contains the memberships for the 
membership groups from Classification. It 
maps a user login to a SID and 1s used to deter- 
mine what resources that login should have. 

UserRights defines owners of classifications and 
grants access to users for a classification. This 
allows a user to manage resources for their 
membership groups. 

ResourceUsedBy defines what resources a classifi- 
cations has. It also indicates if the resource can 
be edited by the owner or proxy of the classifi- 
cation. 

ResourceGroup links classifications together. 


Flexible 


We made an effort to avoid “hardwiring”’ any- 
thing in the system. This was largely accomplished by 
taking a very dynamic view of the data. That is, 
instead of having an ‘‘account” with a “disk quota” 
field that would be assigned a value based on the state 
of the system at the time of the account’s creation, all 
information about an account must go through a pro- 
cess of resolution, where its dependencies are deter- 
mined and the values for each of the account’s proper- 
ties are resolved at run-time. This view was taken for 
everything in the system, not just accounts. 
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Something else that we incorporated in order to 
allow maximal flexibility is a system of property 
inheritance. Although dealing with a relational 
database system at the core, we were able to provide 
the ability to inherit properties from a parent by speci- 
fying a relationship between various records in the 
database by defining a “parent ID” (Called “‘PID’’) as 
a means of determining which “SID” is one step 
closer to the root than the current. 

This gives us the ability to specify object-ori- 
ented “‘is-a’’ relationships between records in the 
database. Thus, the “‘tree’’ of elements to be resolved 
can be of an arbitrary depth, allowing each site (and 
each particular type of resource being managed) to 
have an appropriate number of levels to support the 
sort of abstraction desired, without forcing some high 
level of overhead on those who do not need to deal 
with such abstractions. 


A good example of how we use the ability to 
inherit properties is in the case of a series of courses 
that a student will take in sequence. There will be 
some Classification table entry that will identify the 
series. Each course in the series will have its own 
Classification entry whose PID identifies the Classifi- 
cation entry of the series as the parent. Each section in 
a course will have its own Classification entry whose 
PID identifies the Classification entry of the parent 
course. Thus, configuration changes are made at the 
appropriate level: those that affect all sections in the 
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Figure 1: Schema of Pelendur’s Database 
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series will be made in the Classification entry for the 
series, those that affect all sections in a specific course 
in the series will have those changes made in the 
course’s Classification entry, and those that are spe- 
cific to a particular section will be made in that sec- 
tion’s Classification entry. When we need to deter- 
mine how much quota, for example, an account has, 
we’ll determine which sections in which the account 
has membership. Those will be resolved by walking 
up the tree until we get to the root object (as deter- 
mined by having a PID of 0), populating the fields in 
memory with the values in the database and returning. 
By the time the initial Classification entry retums, we 
will have queried each level in the tree, populating the 
object with the levels specified at the highest level 
first, and overriding those with whatever (if anything) 
was specified in the lower levels. 


Additionally, where values are numeric, they 
need not be absolute; signed numeric values indicate 
relative values. Consequently, a property of a Classifi- 
cation might be to increase disk quota by 30MB, 
instead of specifying some absolute value. 


Modular 


Rather than requiring code changes in many dif- 
ferent places in order to create new interfaces in the 
system, we designed the system to be made up of a 
basic core which includes the database and the 
resource resolution logic. The rest of the system inter- 
faces to that core. The core understands when an 
account needs to be created, or what the properties of 
an account at any given time should be, but it knows 
nothing about creating accounts. Instead, it can tell 
what the account’s properties are and what system(s) 
need to reflect those properties. If the account needs 
access to both Unix and Sybase, the core will pass the 
appropriate information to the modules for Unix and 
sybase through a standardized interface. 


As described in section on implementation lan- 
guage, this design makes it possible for even the 
database itself to be replaced with another database 
that has the same schema. 


Implementation Language 


Perl was chosen as the implementation language. 
Its freely available DBI (database interface) and DBD 
(database driver) package for talking to a wide variety 
of databases makes it an excellent option for building 
atop a database: 

e Very little investment is made in an interface to 
any particular database (since we code to DBI), 
thus allowing us (theoretically) to use any 
database for which a DBD module exists. In 
practice, we did use one Sybase-specific fea- 
ture, which we can and will in the future stop 
using. 

e We can spend our time focusing on the problem 
at hand, instead of how to write to the database. 

e Being free, the price is right. 
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e Since source code is available, if there is a 
problem with it for which a fix is not available, 
we can fix the bug ourselves. 


Because parts of this system will need to run 
with superuser privileges, we’re concerned about 
safety of our code. Perl has some excellent safety fea- 
tures: namely, the ability to identify “‘tainted”’ data 
and support for arrays and buffers that grow dynami- 
cally. 


Finally, Perl is available on a huge number of 
platforms. As long as we keep portability in mind, Perl 
will provide us all of the language support necessary 
to allow our code to run unmodified on essentially any 
system we could use in the forseeable future. 


Programs 


rosterload is the program that loads rosters into the 
database. In our environment, rosters are course 
rosters that identify which students and instruc- 
tors should be associated with a given section. 
For the most part, rosterload just calls Ros- 
ter::rosterload, which does all of the work. Our 
plans are for rosterload and Roster.pm to be 
modified to be able to load roster information 
directly out of the Data Warehouse or from 
email. 

makeaccounts is the program responsible for creat- 
ing things, that 1s, resources and accounts. First 
it will initialize the resource methods, then it 
creates any resources that don’t already exist. 
Then it creates any accounts that don’t exist. 
We have a Cron job run makeaccounts runs twice 
per day. 

expireaccounts is our garbage collector; it’s respon- 
sible for removing things from the system. 
Expired resources are removed from both the 
database and from the systems that were used 
to support the account. After this has been 
accomplished, it will remove entries in the 
‘““MembershipIn”’ table that have expired and 
then any user resources that are specified for 
the user in the database. cron runs expireaccounts 
once per day. 

notifier sends account removal notifications. All 
accounts scheduled to be removed will receive 
a seven day notification. Account that are ‘per- 
sistent’ will also receive a 30 and 14 day notifi- 
cation of expiration.’ Note that this includes not 
only accounts for operating systems, but this 
also includes “‘accounts”’ in software systems 
like Sybase. As Pelendur continues to help us 
blur the distinctions among different systems 
that comprise the “computing environment’, 
we'll move away from providing notification 
that specific systems’ accounts will be unavail- 
able and provide notification only on the user’s 
‘“‘meta-account”’ in the department. The Sybase 


'The amounts of time on the notification are configurable 


parameters. 
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account will be created as soon as the student 
shows up on the roster for a class that has this 
resource. The account will always? match the 
Unix login name and will have the typical 
default password. 

crconfig is the course configuration interface for 
instructors. This provides a convenient means 
for them to manipulate the database in a con- 
trolled manner, allowing them to change only 
that which is under their administrative author- 
ity without creating artificial restrictions that 
require staff to perform their administrative 
tasks for them. Instructors and their ‘“‘proxies”’ 
(those whom they designate) to add and to 
remove resources from a particular section and 
a course default. Project and course directories 
are also configured through this interface. 

cradmin is the account management administration 
tool. Essentially, this is the interface that 1s used 
to manipulate the state of the database on any- 
thing that is in the database. Where crconfig 
deals with abstractions and has a “‘course-spe- 
cific” view of the world, cradmin allows manip- 
ulation of non-course-related resources. The 
user interface itself still works with many of 
these abstractions, but it is in this program 
where we can make changes to the user inter- 
face to allow administration of new resources. 


Modules 


Here we describe the major modules of the Pelen- 
dur system. Many of these modules are shared by vari- 


ous standalone programs in the system. 

IICFDB.pm is the module contains the generic rou- 
tines for interacting with the account manage- 
ment database. There is typically a routine for 
each table for searching, adding, and removing. 
They follow the naming convention get_stable- 
name>, add_<table-name>, remove_<table-name>. 
Table names are converted to lowercase and 
underscores are used where there would be 
white space (““MembershipIn”’ becomes mem- 
bership _in}). For some tables an update function 
exists also. The get_ routines are all polymor- 
phic and will do different searches depending 
on what data is passed to the routine. This mod- 
ule also contains the routines to walk the 
database recursively (resolve_pids) in order to 
resolve all dependencies and provide an up-to- 
the-second view of an account’s properties, as 
determined by what the database knows about 
the account. 

IICFLog.pm is our logging mechanism. Presently, 
this basically accepts messages and puts them 


2 Always” iS a pretty strong word. There are a few ex- 
ceptions, as the result of ancient accounts that predate Pe- 
lendur that still contain dashes (-) so they can’t be used as 
logins to Sybase. If the username has a dash in it, the dash 
will be converted to an underscore (_) for the Sybase ac- 
count. New accounts are always created with names that 
are portable across our systems so that these kinds of con- 
versions will not be necessary. 
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in the “night place’’, but the intention is that it 
will be a general-purpose log gatherer for all 
applications in our environment. 

IICFLogin.pm contains the routines that deals with 
logins in our environment. It contains the rou- 
tines that generates new usernames and default 
passwords. 

NT.pm contains the routines to do all things on the 
NT systems. It is currently not implemented; an 
older standalone account creation and deletion 
system was developed locally for NT accounts. 
Instead of implementing this part of the system 
initially, we opted to focus on the Unix, Sybase, 
and core database portions of the system, build- 
ing an interface between Pelendur and the old 
NT account management scripts. This decision 
has tumed out to work relatively well for us, 
and has saved us from what could have become 
a large amount of redundant work as Microsoft 
seems to think that making major interface 
changes from version to version of its operating 
systems is an appropriate thing to do. Thus, we 
can avoid writing most of NT.pm until the NT 
based: systems are on a newer version of the 
operating system than the current programs 
support. 

PQuota.pm interfaces with the printing system’s 
quota handling. We use LPRng [6] for our 
printing system throughout the department. 

ResourceMethods.pm is a module that contains all of 
the methods for the resources in the account 
management system. Each resource has its 
method defined for it in the METHOD field in its 
“Classification” entry. When called, this 
method will ‘do the right thing’ for the 
resource. For example, a resource like “‘mailing 
list” might have a method that specifies a pro- 
gram to be run in order to create or to delete the 
mailing list. 

Roster.pm contains the routines for processing the 
roster. 

Sybase.pm contains all of the routines for dealing 
with Sybase resources. 

Unix.pm contains all of the routines for Unix 
resources. It adds and removes accounts in the 
password file, manipulates the group file, and 
generates the Quotas file for disk quotas. 

mkcisdir.pm is used to create and to remove directo- 
ries on the Unix systems. It handles several 
types of directories: users’ home directories, 
group project directories, and the directories 
used by Submit, our program for electronic lab- 
oratory submissions. Where possible, this mod- 
ule will run as the owner of the directory 
instead of root. 


The Effect of Pelendur 


Our environment has benefited tremendously 
from Pelendur in many ways. What used to be a 
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painful experience for all is now essentially a non- 
event. 


Labor 


The total staff labor expenditure for processing 
accounts between terms is greatly reduced. 

e Accounts to be removed are no longer pro- 
cessed manually. When an account has no more 
references (managed through the Membershipin 
table), the account 1s garbage collected. 

e Instructors would manually add and remove 
students from their sections using hardcopy 
provided by the university registrar. We now 
get these data from the registrar directly and 
automatically add and remove students. 


Error Rate 


Historically, this has been a problem. The old 
software had to run by someone who knew its idiosyn- 
crasies and limitations. Mistakes were frequent and 
could easily require several hours to fix. Errors are 
now much less frequent, because we get data directly 
from the registrar and do not require any manual inter- 
vention before account configuration. Because the 
entire state of the system is driven by the database, 
errors can now be fixed by making appropriate 
changes in the database and waiting for Pelendur to 
propagate them. 


Since managing our systems with Pelendur, we 
have been able to identify accounts that have expired 
long ago but were never removed, to identify prob- 
lems that arise because of an account being misclassi- 
fied, and generally to free ourselves of the kinds of 
concerns that come about when the administrators 
need to manage things manually. 


Latency 


In section “Programs”, we identified which 
parts of the system run on a regular basis in our envi- 
ronment. These are configurable to a site’s require- 
ments. In our environment specifically, this means that 
changes made take no more than one day to take 
effect. This is a huge difference from the days, weeks, 
or even more that it took under the old system. 


Future Work 


Quite a lot can still be done with Pelendur. 
Specifically, we need to increase the number of sys- 
tems against which we can interface, including native 
NT account management, more intelligent 


Conclusions 


Account management can be greatly simplified 
by taking a more abstract view and thinking of system 
access as a property that results from the state of the 
account. Pelendur has proven to be a highly effective 
means of managing a very large number of highly 
variable accounts. 


Availability 


Although the system has been designed and 
implemented in a way that emphasizes flexibility and 
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freedom from very a very site-specific view of the 
world, it will still take quite a bit of effort for another 
site to bring Pelendur into production. We’re currently 
working on finishing the functionality and hope that 
we will be able to revisit some of the areas of the sys- 
tem that work for us but would make it difficult or 
impossible for other sites to use the system as-is. This 
work is geared toward making a general release of the 
system. We have no idea when this could possibly take 
place. 
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Network Information Management and 
Distribution in a Heterogeneous and 
Decentralized Enterprise Environment 


Alexander Kent & James Clifford — Los Alamos National Laboratory 


ABSTRACT 


To promote enterprise-wide information and resource sharing, we have implemented a 
network information management and distribution system that gives subscribing systems real-time 
access to relevant information changes. Participating systems need little more than a small, single 
application to receive the updates. User interaction and data administration require only a web 
browser. The resulting system is timely, reliable, secure, easy to support and maintain, and 


extensible. 


Introduction 


In Los Alamos National Laboratory’s distributed 
and heterogeneous computing environment of more 
than 12,000 users and more than 20,000 computers, 
the task of maintaining accurate network services 
information is a complex undertaking. Information 
such as network accounts, e-mail addresses, login 
names, aliases, home-page pointers, privileges, per- 
missions, and so on, is constantly changing as people 
come and go or change job assignments. Matters are 
further complicated when the input data comes from 
different sources and must then be distributed across 
multiple server systems. To simplify the system 
administrator ’s job, we designed a system to make it 
easier for network services to get the current and accu- 
rate information they need to do their job. We call it A 
Real Time Information Management and Update Sys- 
tem (ARTIMUS). 


If you have attended past USENIX LISA confer- 
ences or read the proceedings, the problem and solu- 
tion may sound vaguely familiar. The general require- 
ments for an account management system have been 
nearly the same for over ten years, including: [2] 

e Vendor and service independence 

e Data flexibility 

e Minimal changes to existing software 
e Automated account installation 

e User access to data 


This paper focuses on the unique aspects of the 
Los Alamos National Laboratory (LANL) approach. 
Specifically, these areas include the database design, 
manipulation of data within the central database, prop- 
agation of that data to end-hosts and services, and 
security. Recent systems differ on whether they use a 
‘“big hammer’”’ relational database or not. ARTIMUS, 
[1], [4], and [11] do; [2], [7], [9], and [10] do not. 


Motivation 


Our organization, LANL’s Network Engineering 
Group, is the Lab’s Internet service provider. We 
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provide the usual services: LAN and dialup connectiv- 
ity to the Intemet, DNS, white pages directory infor- 
mation, authentication, e-mail accounts, web page 
publishing, and so on. Other LANL organizations 
offer compute, storage, print, personnel, training, 
library, and many more network services to internal 
and remote customers. In addition, organizations 
throughout the Laboratory run their own network 
servers to satisfy local needs. Each organization, and 
even individuals, choose the type and configuration of 
computing system(s) that best meets their needs. 
While this looks like a recipe for chaos, it also pre- 
sents opportunities. 


The group needed a sane way to manage people, 
accounts, and other network information for our ISP 
business. With the right design, we saw that we had a 
chance to help the other network service providers too. 
Subscribers could participate in the same centralized 
account and information management system that we 
use on their own servers. These are the requirements 
we felt we had to meet to have a viable system: 

e Each user should be able to make changes to 
his or her own network information. A third 
party, like a system administrator, should not be 
needed to make the change. 

The end user interface should be intuitive and 
consistent. Managing your network profile will 
not be a frequent or familiar activity. The goal 
here is to allow changes to be entered quickly 
and accurately without a call to the help desk. 
Existing subscriber system software should not 
require modification. Programs, either shrink 
wrap or open software, will still run using 
existing data formats and access methods. 
Additional subscriber systems should be easy to 
integrate. We will not own or manage many of 
these clients. Changes to subscriber systems 
must be few and minor in nature. 

Subscriber systems should not be affected by 
changes to our data structures and business 
rules. 


85 


Network Information Management and Distribution... 


e Subscriber systems should not be affected by 
failures in the information management system. 
e Subscriber systems should be notified of data 
changes in near real-time so users can view and 
test changes immediately after making them. 
Subscriber systems should get only accurate, 
consistent, and current data. 
System administrators should retain control 
over their systems. For example, they should be 
able to control who gets access and privileges 
on their systems. There should also be provi- 
sions for system administrators to make local 
changes outside of ARTIMUS. 
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Figure 1: Example data flow for a user setting a new 
e-mail forwarding address for a given name, and 
sendmail requesting the forwarding address. 


e System administrators should see their work- 
load drop after participating in ARTIMUS. 

e ARTIMUS should be extensible. The difficulty 
in adding new services and operating systems 
should be in step with their complexity. 
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e ARTIMUS should have high availability. 

e ARTIMUS should import corporate data to 
eliminate redundant data entry and promote 
consistency. For example, employee data from 
HR could be used to build organizational e-mail 
lists. 

e ARTIMUS should be secure. Information 
should be readable and modifiable only by 
authorized persons. 

e ARTIMUS, not subscriber systems, should 
implement site policies and business rules. 


Overview and Example 


Here is a simplified example to demonstrate 
ARTIMUS’ data flow, which can be seen in Figure 1. 
Suppose Amy wants mail that 1s sent to the institu- 
tional address help@lanl.gov to be delivered to her 
departmental server account amy@cic-mail.lanl.gov . 
Here 1s how it is done: Amy uses her web browser to 
go to the Network Registry service and logs in to iden- 
tify herself. She creates help as a new name which the 
Network Registry inserts into the database assigning 
ownership to Amy. Then she sets the forwarding 
address for help and the web server completes the task 
by up dating the database. The change to the table con- 
taining names triggers a database procedure that 
makes a copy of the updated record and notifies the 
Trigger Event Daemon (TED) process (see the section 
“End Service Propagation’’). TED notifies the sub- 
scriber services affected by the forwarding address 
change so those systems can update their local copies 
of e-mail forwarding information. 


System Design 


ARTIMUS 1s a three-tier design of web-based 
user interface to central database to end-system propa- 
gation. The system is functionally divided into these 
three distinct segments to provide maximum security, 
flexibility and simplicity. 

Web Interface 


Users view and change information with 
ARTIMUS’ web interface called the Network Reg- 
istry. All user modifiable information is updated in 
this manner, traditional applications are not used (e.g., 
chfn). This GUI interface is designed to be friendlier 
than native system interfaces. In addition, the SSL 
web interface encrypts the data between the desktop 
and the web server thus protecting the registration 
information from capture and modification. 


User access to ARTIMUS data is controlled 
through the web interface. The user must first authen- 
ticate to the Network Registry with an employee iden- 
tifier and password. The web interface then enforces 
access controls that restrict what the user can change 
within the database. Normally users may only change 
information they “‘own.” However, system adminis- 
trators may be granted access to change other users’ 
information. We considered pushing the authentication 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Kent & Clifford 


and access control into the database itself but Sybase, 
which we currently use, does not support any appro- 
priate external authentication systems like RADIUS or 
Kerberos. 


In the previous example of Amy changing the 
forwarding address of help@lanl.gov, Amy goes to the 
URL for the Network Registry and authenticates with 
her employee identifier and password. Selecting the 
name help she then enters the forwarding address as 
seen in Figure 2. The Network Registry then does a 
SMTP VRFY on the given forwarding address and 
submits an update to the database name table if the 
address is valid. 


In addition to e-mail forwarding, the Network 
Registry is us ed to create and remove user accounts on 
UNIX and Microsoft-based systems, change pass- 
words, and manipulate several other user network 
attributes. Additional features and systems are con- 
stantly being added to the Network Registry. 


The web interface is built upon whiz [8], a 
Python-based sequential CGI forms tool that provides 
a simple, stateful, and authenticated method of manag- 
ing the registry segments and adding additional fea- 
tures. Communication to the database is handled 
through a locally developed system called sqld which 
wraps database traffic with SSL-based encryption 
between the Network Registry and the database. 
Other, more standard database access libraries could 
be substituted for sald. 


The Database 


The authoritative copy of nearly all network ser- 
vice information is kept in a relational database. The 
Network Registry and various corporate repositories, 
like human resources’ employee data, are ARTIMUS’ 
information sources. The database enforces syntax, 
consistency, various business rules, and serves as a 
central information clearinghouse. 


User Registration 


Add Account 
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If the information is accurate in the database, it 
will be accurate on the servers. And if it is wrong in 
the database, it will be wrong everywhere. Because 
data integrity is so important, we use the database’s 
features such as syntax checking on table fields, inter- 
table consistency enforcement, and “‘triggers”’ that call 
stored procedures to validate record changes. Both 
Sybase and Oracle products support these features. 
Previousl y, we tried maintaining consistency by doing 
the checking in the web server and other applications 
that modified the database. It worked but changes to 
table structures and field definitions were more diffi- 
cult because several programmers had to modify code 
simultaneously. This explains our move to a “big 
hammer” relational database. 


The database tables are divided into three pri- 
mary categories: one set for person data, one for name 
data, and one for authentication/authorization data. 
Person data is needed for ownership; 1.e., some person 
owns each table entry in the database. Person data is 
also needed for white-pages publishing. A name has 
an owner and may have attributes such as as a for- 
warding address, a URL, a UNIX UID, sharing group 
members, and mail list members. Authentication and 
authorization data include a persons passwords, 
authentication tokens, accounts, and permissions or 
authorizations. 


A portion of the network information database is 
shown in Figure 3. Designing the database tables was 
a non-trivial, iterative process. Subscriber require- 
ments for data, syntax, and business rules were gath- 
ered and database design principles were applied [6]. 
The resulting design was reviewed by peers and cus- 
tomers. Then the process was repeated as more ser- 
vices were added and mistakes were uncovered. 


Here 1s how the syntax, integrity, and business 
rules are applied in Amy’s e-mail forwarding address 
example. Before help is added to the name table, the 


E-Mail Options -- Set Forwarding Address 


Choose the name you wish to forward e-mail for: 


Set Forwarding 
Address 


Change Mail 
Server Password 


E-Mail List 
Manager 


Delete Account 


[ help@lanl. gov x] 


Enter the forwarding address: 


[amy@cic-mail.lanl. gov | 
Next --> | 


Figure 2: Network Registry page for adding/changing e-mail forwarding addresses. 
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Figure 3: Major tables and dependencies within the database. 
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length and characters are checked. Names with @’s or 
spaces are rejected.! Also, names that already exist are 
rejected to enforce uniqueness. 


hosts 


- 


DPR es 
nosmame — 


-admin_ email 





services 


hostname (FK) 
table_name 





service LDAPinetLocalMailRecip 


hostname (FK) 
ID 
pass 


table_name 
action 
name 
email 





Figure 4: TED support tables within the database. 


Similarly, the e-mail forwarding address, 
amy@cic-mail.lanl.gov, is checked before it is added 
to the database. This time an @ is required. 


The effects of applying the concepts of business 
rules and data integrity are more striking in the exam- 
ple of an employee leaving. Our first rule is to wait for 
seven days after someone’s personnel record disap- 
pears before deleting any data. Data entry mistakes 
happen and resurrections are simple.2 When a person- 
nel record disappears, that individual’s expiration date 
is set to today plus seven days in the person table. 
Later, a cron job initiates deleting expired people. 
Before a person is deleted, the database deletes all of 
his or her names. The SQL command delete from name 


‘Names cannot have a @ because an implicit @lanl.gov is 
Beenie when associated with a forwarding address. 
While not actually removing any infor:nation, all authenti- 
cation and authorization (UNIX, NT, etc.) accounts for the 
person are disabled immediately. 
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where zn=’Amy’? deletes all of Amy’s names. But before 
a name can be deleted, the database also cleans up 
anything that depends on that name. For example, 
when help is deleted, e-mail lists are cleaned up with 
delete from name_list_members where name="help’. This 
cascaded cleaning keeps the database internally con- 
sistent and, even better, reduces bounced e-mail to list 
owners. 


End Service Propagation 


Propagating information from the database to the 
subscriber systems is a critical component of 
ARTIMUS. The in-house written subsystem that per- 
forms this function is affectionately called TED and 
UFRD. 


The Trigger Event Daemon (TED) feeds 
database changes to the Update Fields Real-time Dae- 
mons (UFRD) running on subscriber systems. UFRD 
passes the changes to local programs that then process 
the updates. See Figure 1. 


_ The TED-UFRD subsystem is somewhat similar 
to the Project Athena Zephyr Notification System [3] 
in terms of providing immediate, reliable, and high 
fan-out information propagation. However, the TED- 
UFRD design is simpler and specific to transmitting 
information from a central source, the database, to end 
systems, the subscribers. Data transmission security 
and subscriber system data access control are built in. 


The TED-UFRD system disseminates informa- 
tion changes in near real-time from database tables to 
a list of subscriber hosts affected by the changes. 
Instead of requiring services to wait for cron jobs to 
run or other polling mechanisms, the end services are 
immediately notified of information updates. Oracle 
has a similar notification facility but requires Oracle 
software on each subscriber system [5]. 


Currently LANL updates the following services 
through TED-UFRD with an average of 1500 informa- 
tion events per day: 

e RADIUS dial-in passwords for the institutional 
modem pool (1150+ users) 

e CRYTPOCard one-time password token card 
accounts (7200+ users) 

e Microsoft master accounts domain for the insti- 
tution (4400+ users and global groups) 

e LDAP white-pages system (19,000+ record 
objects) 


Additional services will begin using TED and 
UFRD over the next few months, including UNIX 
account and group management, Kerberos, Entrust, 
and DCE/DFS systems. 


Trigger Event Daemon 


Moving data from the database system to TED 
starts with triggers on tables containing information 
needed by the subscriber services. In the e-mail for- 
warding address example, the trigger is on the name 


table. A trigger fires (1.e., calls a stored procedure in 


3zn is a unique person identifier, usually an employee num- 
ber. Amy is used here for simplicity. 


89 


Network Information Management and Distribution... 


the database) when a table is changed. The database 
allows one trigger per action (add, update, and delete) 
per table so TED triggers must coexist with a table’s 
integrity triggers. 


create trigger name_trg 


as 


On name> For a2nsert 


-- Begin TED update 


declare @tbl_name varchar (40) 
,@name typ_name 
,@zn CY P21 
,@reserved_name typ_flag 
»@email typ_email 
,@URL Varchar (255) 
»@login_name typ_flag 
»@location varnehnar (4:6) 
»@pager typ_pager 
,»@phone typ_phone 
»@fax typ_fax 
,@uc_ms varchar (4) 
,@info varchar (255) 
»@expiration datetime 
»@ntmad_group typ_flag 

select @tlb_name = "lanl..name" 


declare. ted crsr cursor 

for select. * from inserted 
open ted_crsr 
ted_loop: 
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To remember table changes, TED triggers create 
new records in TED service tables with the type of 
modification (add, delete, delete-update, or add- 
update), the name of the table being modified, a 


fetch ted_crsr into @name,@zn,@reserved_name,@email,@URL, 
@login_name,@location,@pager,@phone,@fax,@uc_ms, 


@info,@expiration,@ntmad_group 
if @@sqistatus = O begin 


exec TED..send_service_LDAPmail_fwd(@tbl_name, 


"ADD" ,@name, @email) 
goto ted_loop 
end 
elose ted2crsr 


-- End TED update 
Figure 5: Insert trigger attached to name table for updating e-mail addresses. 


create proc send_service_LDAPmail_fwd 


as 


@table varchar (40) 
»@action cCyplaetvon 
,@name typ_name 
»@fwd_addr typ_email 


declare: host. ecrsr cursor 


for select hostname from TED..services 
where service_name=@service_name 


open host_crsr 
host_loop: 
fetch host<crsr- ante @host 
if @@sqlistatus = O begin 
insert LED: Service jnixsr v 


values ( 


@host,0,@table,@action, @name, @fwd_addr) 


goto host_loop 
end 
elose “host crsr 


exec TED. .notify("LDAPinetLocalMailRecip") 


return, 0 


Figure 6: Stored procedure called by triggers attached to name table to move records to TED tables and notify TED 
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subscriber host name, and the modified fields. A sepa- 
rate record is inserted into the service table for each 
subscriber host needing the information. The service 
table for e-mail forwarding and two other TED sup- 
port tables are shown in Figure 4. The trigger sends a 
UDP packet to TED with the name of the service table 
where the inserts were placed. To simplify the coding, 
TED’s triggers actually call a stored procedure to 
insert the records into TED service tables. The trigger 
for e-mail forwarding attached to the name table can 


#t/usr/bin/perl 

use Net::LDAPapi; 

use Sys::Syslog; 

## Update inetMailRecipient object. 
df Changes read from stdin look like: 
# table|action|name|fwd_address 
slocal domain, —" lanl gov" 
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be seen in Figure 5. The stored procedure called by the 
trigger is in Figure 6. 

Upon receiving the UDP packet, TED reads the 
indicated service table from the database.* Then TED 
sends a UDP packet to each subscriber host it finds in 
the service table records. The UDP packet signals the 
host(s) to start UFRD. The host’s UFRD makes a TCP 


‘TED will work with multiple database servers. As a secu- 
rity measure, it will only connect to database system(s) in its 
configuration file. 


Ssep=chr(255): # Separate fields with char 255 
SROOTDN = “cn=root, o=Los Alamos National Laboratory, c=US"; 


SROOTEN = “oob"; 
Sidap server. = “idap: lanl soy: 


&openlog("ufrd.LDAPmailRecipient", pid’, ’daemon’ ); 


&Sys::Syslog::setlogsock(’unix’); 
Sld = new Net::LDAPapi($ldap_server) ; 
if: “Cold <==" 214 


&syslog("err","Connection to $ldap_server failed") ; 


exit 1s 


} 


Me CS dO bands (SROOTDN;sROOTPW): -'= LDAP SUCCESS yt 
&syslog("err. "LDAP bind error: Sld-verrstring™):; 


excdet als 
J 
wot le (<P) 4 
chomp; 


(Stable,Saction, $Sname,$fwd_addr)=split(Ssep):; 
SENTRYDN="cn=Sname, objectClass=inetLocalMailRecipient"; 
## Treat update add/deletes same as regular add/deletes 


if. (Saction: ="“"DELETE™ |||) Saction =" 


PUPD Eas 4 


if ($ld->delete_s(SENTRYDN) != LDAP_SUCCESS) { 
&syslog("err","LDAP delete error: $ld->errstring"); 


exit 1; 
} 
Feeilsat(Saetion =~ "ADD" ||| Saction 
%ldapdata = ("cn" => Sname, 


"UPADD") { 


"objectClass" => "inetLocalMailRecipient", 
"mailLocalAddress" => "Sname==([@]==S$local_domain", 
"mailRoutingAddress" => S$fwd_addr) ; 


if ($ld->add_s (SENTRYDN,%ldapdata) 


{= LDAP SUCCKESS) 4 


&syslog("“err™, "LDAP add error: S$ld-=Perrstring") ; 


exist ls 


} 
} 
S$ld->unbind: 
exit O; 


Figure 7: UFRD client that creates/modifies/deletes LDAP inetLocalMailRecipient objects. 
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connection to TED and reads its information updates. 
When a change is successfully processed, UFRD 
sends TED an acknowledgement and the matching 
service table record for that host is removed. A TED 
service table record is kept until an acknowledgement 
is received so failures with UFRD can be retried. 


To handle subscriber hosts that do not request 
their information updates, TED periodically reads all 
service tables’ unprocessed data records. For each sys- 
tem with pending updates, TED sends another UDP 
notification since the subscriber host may have been 
down or unreachable. To summarize, information 
change notifications are sent to subscriber systems in 
near real-time and resent until successfully processed 
and acknowledged. 


Because the information is_ stored in the 
database, state 1s maintained even if TED or the 
UFRD clients should terminate for unexpected rea- 
sons. TED will send an e-mail notification to the 
administrator listed in the TED’s hosts table if a sub- 
scriber has unprocessed requests over two hours old. 


Update Fields Real-time Daemon 


A UDP packet from TED starts a UFRD on a 
subscriber host; on UNIX systems UFRD 1s started by 
inetd. The packet contains a port number to connect 
back to on the TED system. UFRD first reads its con- 
figuration file to find the encryption key to use with 
the TED system that sent the UDP packet. UFRD then 
makes a TCP connection to the TED server and sets 
up DES3 encryption using the key it found. UFRD 
receives the pending information updates and, based 
on the table name field, executes the appropriate com- 
mand to process the data. The information update 
line(s) are passed to the command as standard input 
under UNIX and message passing under Windows NT. 
If the command exits successfully, UFRD indicates to 
TED to remove the corresponding record(s) from the 
TED service table. 


The commands called by UFRD to update infor- 
mation may be written in the most convenient method: 
C, C+4, Perl, Python, shell script, or some specialized 
interface for the service. The Perl code to process e- 
mail forwarding address changes 1s in Figure 7. 


UFRD has been ported to several operating sys- 
tems including Linux, Solaris, UNICOS, Irix, and 
Windows NT. Its simple functionality relying on run- 
ning native commands to update local data allows 
quick porting to new platforms. 


Future Plans 


The group has several future improvements 
planned for the ARTIMUS system. 


Extending ARTIMUS to other network informa- 
tion systems is foremost. Generating dynamic DNS 
updates is a goal for the next year. Our host informa- 
tion 1s already maintained through a web interface and 
kept in a Sybase database but does not use ARTIMUS. 
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The difficult part will be reconciling the names’ own- 
ers. 


Offering the service to other Laboratory organi- 
zations has begun and will be expanded. Given the 
flexible and secure ARTIMUS framework, it 1s easy to 
provide central account management services to net- 
work servers throughout the Laboratory. Adding a 
Network Registry module, a few database tables, con- 
straints, and triggers, as well as writing an update 
command for UFRD service to call, is not difficult. 


The Network Registry continues to grow. Cur- 
rently, we are redesigning much of the look and feel in 
attempt to make it easier and more intuitive for the 
end-users; user interface design is non-trivial. Various 
business and integrity checks continue to be added, 
both within the Network Registry and the database. 


ARTIMUS needs additional rules to prevent 
inappropriate bulk changes from happening. Occa- 
sionally, we receive a large, but bogus, data feed from 
the corporate data warehouse that must be detected 
and rejected. 


We would like to get real-time updates propa- 
gated from the Lab’s corporate sources to pass along 
to end customers. Such real-time linkage would allow 
new employees to request accounts upon arrival (and 
have them created in real-time as well, thanks to 
ARTIMUS), plus automatically disable authentication 
and authorization accounts the moment they are termi- 
nated in the human resource system. 


We would like to examine the use of individual 
user accounts within the database and direct authenti- 
cation to the database. The tradeoffs of security bene- 
fits versus complexity may or may not make this prac- 
tical. For various security issues we are considering 
moving from Sybase to Oracle and using Oracle’s 
Kerberos and RADIUS authentication. Oracle’s 
DBMS ALERT package may also be superior to the 
current UDP-based database notification mechanism 
for TED. 


Additional issues, ideas, and plans will come up 
as we add new services and think up new methods to 
use the system. 


Conclusions 


The ARTIMUS design, including its web inter- 
face, relational database with built in data integrity, 
real-time update service, and data protection provides 
a solid base for eventually managing nearly all of the 
Laboratory’s account and network information data. 
We are optimistic the implementation will meet or 
exceed the requirements listed in the Motivation sec- 
tion and discussed below. Ultimately, we will measure 
the system’s success by the number of subscribers it 
supports. 


User Interface Requirements 


Users can manage their own network informa- 
tion through the Network Registry. Web based 
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applications with authentication are commonplace at 
Los Alamos. Most customers will be able to easily 
maintain their own network profiles. Keeping the nav- 
igation simple, intuitive and consistent as features are 
added will be a challenge. 


Subscriber System Requirements 


Adding subscriber systems to ARTIMUS 
requires new entries in the TED tables, a UFRD dae- 
mon, a UFRD service application, a configuration file, 
and a shared DES key. The total installation time 1s 
usually under an hour. 


Currently, 99 percent of the changes to the 
database are propagated to the subscriber systems in 
under two seconds via TED-UFRD. We expect the 
mean response time and standard deviation to both 
shrink when we retire some frequently run ad hoc 
database query applications. 

System Administrator Requirements 

There are a few system administrator features in 
ARTIMUS that are outside the main theme of this 
paper. We will briefly mention them here. Common 
system names like “root” are reserved, as are UNIX 
UIDs and GIDs under 1000. OS, third party, and local 
software that define users and groups can coexist with 
ARTIMUS. System administrators can define their 
subscriber systems as open or closed. Users can add 
accounts to open systems but only system administra- 
tors or their designees may add accounts to closed sys- 
tems. Finally, we have defined organization adminis- 
trators who can manage most information for anyone 
in an organization, and name administrators who, in 
addition to the owner, can change any of a name’s 
attributes. 


ARTIMUS Requirements 


ARTIMUS already supports UFRD on a variety 
of operating systems (Figure 8) and services (Figure 
9) but not all services are supported on all OSs. The 
first UFRD for a UNIX system took about a month to 
write and debug. The first UFRD for Windows NT 
also took about a month. Bringing up subsequent 
UNIX UFRDs 1s proportional to the time it takes to 
locate the necessary system include files and libraries. 


E-mail forwarding addresses (aliases) 
URL’s 
UNIX accounts 
UNIX UID and GID’s 
LDAP white pages 
LDAP POSIX users and groups* 
CRY PTOCard token cards 
Kerberos accounts* 
| Authorizations* 
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UFRD agent scripts as seen in the LDAP e-mail 
address example (Figure 7) are usually simple and 
straightforward to write. The approach is: read stan- 
dard input; split the line into an array (or list); and 
execute some existing application to add, delete, or 
modify some local data file. A Perl program to man- 
age Linux user accounts by calling adduser, deluser, 
and moduser is 36 lines long. 


Microsoft Windows NT Microsoft Windows 2000* 


Linux Sun Solaris 
SGI IRIX Compaq Tru64 UNIX* 
IBM AIX Cray/SGI UNICOS 


* under development 
Figure 8: Operating systems where UFRD runs. 


The current ARTIMUS implementation 1s reli- 
able. We do health checks on our services every few 
minutes throughout the day. When a service 1s down 
for five minutes or more, an on-call person is paged. 
The last six months of web, database, and TED error 
logs show only four transient failures. Connecting to 
the web server failed twice and connecting to the 
database also failed twice. There were no prolonged 
failures resulting in an alarm page. 


Since subscriber systems depend on ARTIMUS 
for updates, downtime is only noticeable to users 
wanting to make changes. Clearly, the main thing to 
avoid is losing the database. Backups and mirroring 
are both done. 


Oddly, overall network service availability can 
actually be improved by adding a central database. 
LDAP and RADIUS are important services to the 
Laboratory. ARTIMUS makes replicating these 
servers as easy as deploying a system and adding a 
hostname to the TED host and service tables. 


Data integrity is a key feature of ARTIMUS. It is 
also the most complex to implement. But, when done 
correctly, it promises the biggest rewards from the sys- 
tem. Our most difficult and tre consuming problems 
are usually caused by incorrect, incomplete, or incon- 
sistent data on one or more server systems. ARTIMUS 


E-mail lists* 

Directories for web and FTP publishing* 
Windows accounts and sharing groups 
LDAP yellow pages* 

RADIUS accounts and passwords 

DCE accounts* 


DNS resource records* 





| SecurID token cards* Entrust PKI certificates* 


* under development 
Figure 9: Applications that currently have UFRD service interfaces. 
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prevents these problems by simply rejecting bad infor- 
mation before it goes into the database. 


Finally, security 1s an essential part of the 
ARTIMUS design. This system would not be 
deployed at Los Alamos if it did not protect users’ net- 
work information. 


Availability 


Due to U. S. encryption export regulations, 
release of the source for the TED-UFRD propagation 
system must be controlled. Those interested in obtain- 
ing the source code may contact the authors directly to 
determine the necessary arrangements. Licensing 
restrictions beyond the export issue are otherwise lib- 
eral. 


The web pages and database currently include 
many LANL idiosyncrasies. They would probably 
require significant changes to work elsewhere and 
therefore are not available at this time. 
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ABSTRACT 


The xps program dynamically displays the Unix processes as a tree or forest in an X Window, 
the roots on the le ft and the leaf processes (those with no children) on the right. The status of each 
process running, sleeping, stopped, etc., can be indicated by different colors. Different users can 
appear as different colors too. 


Process selection can be made per user, all users, or through a regular-expression pattern. 


In contrast to the terminal-based pstree or tree-widget based programs, the tree display uses 
diagonal lines, and effort is made to effectively use the full 2-dimensional area of the screen by 
balancing levels and centering the children of a node between their parent. A goal of the program 
is to give an idea of what’s going on graphically as things may be constantly changing. Therefore 
the display algorithm tries to keep processes close to their parents to reduce the amount of 
scrolling to see localized process creation and destruction. Some effort is also given to make sure 
that the tree layout doesn’t get wildly reorganized when there are small or localized changes. This 
makes it easier for the eye to pick up and recognize the changes over a potentially large display 
area. 


We describe here criteria for tree animations such as this one and how the xps layout 
algorithm works. 


There are some other miscellaneous features of xps. One can select viewing the processes by 
a single user, a regular expression for users, by all users, and perhaps show kernel processes. One 
can click on a process to get more information (via ps or a user-specified program) about that 
process, send a signal, or set the process priority, assuming you have the permission to do so. 


Since programs of this ilk can consume a bit of CPU on their own, some effort has been 
made to turn off the update process when the program is iconified or not visible for some other 
reason such as being obscured by another window. Some attention has been paid to make 
algorithm display fairly fast in most situations, although it has to be admitted that this comes 


sometimes at the expense of a nicer layout. 


What Is xps? Why xps? 


There are a number of front ends or GUIs that 
perform various underlying commands, and some- 
times one wants to see what’s going on behind the 
scenes. Some examples include: 

e Learning more about what is going on in con- 
figuration and installation processes; make or 
GNU configure, for example 

e Tracking down zombie creation 

e Understanding the processes for a user or con- 
tained in a process group; these generally clus- 
ter into subtrees 

e Helping distinguish a process via its lineage. 
For example one may have many bash sessions 
running. Some may be spawned from sshd, 
some from telnet, some via emacs or an xinit ses- 
sion. 


To glean what’s going on, many systems admin- 
istrators use ps or {op or a top variant, like gtop. The ps 
program can list the process id and its parent process 
id. However it 1s sometimes useful to display the par- 
ent-child relation graphically. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Considerations for Doing a Tree Layout for xps: 
Tree Animations 


XpS uses a custom tree-layout algorithm for posi- 
tions nodes and draws lines between them using lo w- 
level Xlib calls. Probably each time a new graphics 
library came out I considered ditching the custom tree- 
layout routine with a generic tree package. However to 
date, I’ve not found an acceptable one. 


Tree widgets are often used for browsing file 
systems or menu systems. Generally these objects are 
fairly static. There 1s generally a way to collapse or 
expand a branch in the tree; this makes sense when 
information doesn’t change all that much. But in the 
context of xps, sometimes the new or changed infor- 
mation is precisely what you want to see. So instead, 
in Xps, symbolic or filtering rules are used to focus the 
display, rather than by zooming in or out from a user- 
selected tree branch. In particular, one can select the 
area of interest by a regular expression which is 
matched against the process-owner name. 


In trees with menus or files, the display is pretty 
much static, and therefore the time users spent to 
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customize the display may be cost effective; it may 
not be as important to spend time initially making con- 
nections between tree objects immediately visually 
distinct. Convention or experience seems to indicate 
that it 1s acceptable to use a Manhattan-metric line 
connection style, that is, where tree lines either go hor- 
izontally or vertically. See Figure 1 for a tree layout 
using Manhattan-metric lines. 


emacs —_ bash 
tcsh ____ make pod2html 
make gco __-_=_ icc] 
gec___ as 
&cc _______ Cpp 
tcsh ~~ netscape 


Figure 1: Lopsidedness and space-wasting layout 
using a Manhattan-metric tree widget. 


The tree-layout requirements of xps are perhaps a 
bit more difficult to satisfy. In fact xps might be better 
thought of as an animation rather than a static charting 
program, and I am not aware of much literature on 
doing animations with trees or forests. 


For an analogy, consider the differences between 
viewing a still-life painting and an animated cartoon. 
The cells of an animated cartoon do not have to be as 
perfectly rendered as a painting. Instead it is more 
important in an animation to make the series of cells 
relate to convey a story or action. 


Below we gives a simple example of the kinds of 
problems faced. 


In a world where things don’t change, like a fig- 
ure in a book, most people will find it most it pleasing 
for graphs with one or two children to look as in the 
left-hand part of Figure 2 rather than the left-hand part 
of Figure 3. 


child! 

root root —————._ child 
child2 

Figure 2: Aesthetic tree layout when things don’t 


change. 
root ~~ root —————_ childl 
child2 
aw 7. childl 
root ~————__ child2 root a 


Figure 3: Possibly better tree layouts when trees shift 
between one another. 


However, now consider the case where things are 
constantly changing. Suppose we want to chart what 
is going on when we have a process that forks a pro- 
cess, child1, and then forks a number of other pro- 
cesses sequentially. For a moment there will be child2; 
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it dies, and then suppose a moment later a new pro- 
cesses, child3, is spawned. Suppose that now runs for a 
short period of time and dies; so we are back to the 
single child1; then a new process child4 starts up and so 
on. 


In this scenario, if you render the one or two 
child processes as in Figure 2, you will see a bit of 
flicker. This is distracting and does not assist compre- 
hending what’s really going on: that a new process is 
being started and finished while nothing is happening 
to child1. If instead the graphs were rendered as in 
Figure 3, the display is more pleasing to the eye in 
animation; it is less spastic and more comprehensible 
even though the left-hand or right-hand side in isola- 
tion may not be the most esthetic rendering of the 
graph. 

In sum, hysteresis of layout can help visualiza- 
tion. 


Now consider the differences between a Manhat- 
tan-metric layout of the variety one often sees with a 
tree widget, and one that uses diagonal lines in an arti- 
ficial but not uncommon situation as is often seen in 
xps. 

Notice in Figure 1, how the root of the tree, 
emacs, is very far removed from its bottom-most node. 
This is not a problem with the Manhattan-metric con- 
nection style per se. However almost all most tree 
widget programs do this kind of layout; to set the posi- 
tion of a node, the layout algorithm can be extremely 
simple since it doesn’t have to consider the positions 
of the children. 


In addition to the lopsidedness, there’s a bit of 
space that is wasted between make and netscape that 
doesn’t appear in Figure 4. Furthermore, the blank 
area to the right of bash and pod2html is reclaimed 
reducing the overall dimensions of the graph. How- 
ever, the more-compact layout is at not at the expense 
of readability. 


bash # podZhtml gcc 
emacs < tcsh —— make make < gcc 


icsh —— netscape gcc 





ccl 





as 


cpp 

Figure 4: Layout. Nodes are more centered around 
their children; diagonal lines improve readability. 
Layout makes better use of available space and 
therefore dimensions are reduced. 





Finally, compare Figure 1 and Figure 4 to see 
how the use of diagonal lines helps readability. 


Experience has suggested these display criteria 
for a tree-animation program such as xps: 

e The layout needs to be fast — it is performed 
every second 

e The layout should be compact 

e The layout should have hysteresis — reduce 
flickering which is annoying to watch and 
makes finding important changes hard to spot 

e The layout should be “pretty”’ on the display so 
users can understand the relationships quickly 
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We now go into some of these needs in more 
detail. 


Fast Layout 

In designing any monitoring program, such as xps, a 
cardinal rule should be: don’t trash the system you are 
trying to monitor. 


Because layout needs to be fast and work for a 
large number of nodes, positioning decisions should 
be pretty much linear. Currently one pass from root to 
the leaves is made. I’ve considered making a back- 
ward pass as well — leaves to roots. 


The program has a number of hacks to avoid lay- 
out recalculation when it is not needed. It checks to 
see if processes have changed; if not, nothing is 
changed. However this is tricky since xpsS shows the 
status of each process (e.g., whether it 1s running or 
sleeping or stopped), and if this alone changes some 
redisplay of the node color (not position) 1s needed. If 
the display becomes iconified or fully obscured, no 
layout is computed. 


Pretty Layout 


Experience has shown that it is preferable to 
draw straight lines for parent — child connections 
rather than diagonal lines since diagonal lines take 
longer to draw and can appear jagged unless antialias- 
ing is used. (Antialiasing also generally takes more 
time to compute and draw.) On the other hand, when 
needed, I find diagonal lines are easier to follow than 
Manhattan-metric lines. 


Similarly, I’ve found that lining things up hori- 
zontally and vertically helps. 


Layout Heuristics 


The key to a program like this is its tree-forest 
display algorithm. In the previous section we 
described desirable qualities; in this section we’ll give 
a rough idea of the actual heuristics used. In the next 
section we go over how this is implemented and the 
time complexity of the layout algorithm. 


Trees tend to be narrow at the root and get bushy 
as one moves towards the leaves. Therefore the 
approach used in xps is put nodes into fixed levels 
(which run horizontally) as we move from the roots to 
the leaves (left to right here). 


The algorithm we use pretty much makes one 
pass to keep things fast. We keep track of the maxi- 
mum number of nodes at a given level as we move 
from roots to leaves. As the breadth increases, the gap 
between levels decreases up to a minimum threshold. 
When things get too tight, the overall dimensions of 
the graph mcreases; in display, the tree breadth is 
shown in the vertical direction. 


Figure 5 gives an example of how levels are redi- 
vided as we move deeper in a tree. 


Putting things in levels increases the likelihood 
that parent-child nodes will line up and that they will 
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stay lined up over time. Actually, although it might be 
useful to have xps take into account old positions of 
node as discussed in a previous section, nght now it 
doesn’t; it is just an artifact of the way the layout 
occurs that this tends to happen. 
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Figure 5: Tree layout along maximum-breath levels. 
Dotted lines show the current levels that are in 
effect for layout at subsequent levels. The node 
labeled A has rank 2 and virtual rank 3. 


After creating the levels for a given depth, the 
dotted lines in Figure 5, xps next positions the node for 
that depth. To this end, the program keeps track of its 
level position or virtual rank in that depth, and com- 
pares it to the virtual rank of its parent. The virtual 
rank differs from the number of nodes placed so far 
(or rank), when we’ve decided to leave a level slot 
empty so as to position a child closer to its parent. In 
Figure 5, the node marked “‘A”’ has a virtual rank of 3, 
while its rank is 2. This is because we’ve moved the 
node down to its level slot 3 so it will line up with its 
parent. 


Note that since the number of levels only 
increases, and when this happens inter-level space 
between level generally decreases, if the virtual rank 
of child is less than or equal to the virtual rank of its 
parent, it can be positioned no further down than its 
parent. 


When setting the position of a node, Xps tries to 
position the virtual rank of the middle child close to or 
less than the virtual rank of the parent, but not so 
much that the overall breadth is increased. 


Figure 6 shows what goes on here. In the upper- 
left diagram we show the virtual ranks and before 
readjustment; to the right of that, after the readjust- 
ment at the new depth. In the bottom-left we see what 
happens when the new level is attached to a node fur- 
ther down; since centering the children would increase 
the overall breadth and dimensions of the tree we do 
not center around the middle child. The bottom-right 
graph show a heuristic not employed by xps but might 
be if a backward pass were made: readjusting the par- 
ent. 


There is one layout problem that should be men- 
tioned and is shown in Figure 7. 


We see here that in drawing lines from the end of 
a node, nodes with a short name can sometimes cross 
over a neighboring node name. Currently xps does not 
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try to prevent this from happening. We have experi- 
mented with doing a layout as in Figure 8 without 
much success. Most of the time, though, the lines do 
not go through process labels. 


Layout Algorithm implementation 


The program needs to do a topological sort. A 
depth-first search is done to assign depth number. 


The roots are then sorted by uid and pid. There 1s 
generally a small number of roots, often one. At 
depths deeper than the root, nodes are first arranged 
by attachment to a parent. 


Within the children of a node; sorting is done by 
a bubble sort. Many people with computer science 
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training cringe when they hear this, because they have 
been taught a bubble sort takes O(n”) time in the worst 
case, while there are many sorting algorithms that take 
O(n log n) time in the worst or average case. However, 
a bubble sort has an advantage our most sorting algo- 
rithms: when used on almost sorted data, the running 
time 1s linear. 


In the context of xps, the processes are often 
pretty much sorted by process id. The bubble-sort 
code we use does a check to stop early when the items 
are fully sorted. Also, it should be noted that the num- 
ber of items to be sorted is often relatively small, since 
the sort is performed only on children of a node. A 
bubble sort generally has less overhead than most 
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Figure 9: Aesthetic tree layout when things don’t change. 
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other sorting programs. Still, a check on the number of 
children should be made and a sort like quicksort 
could be used if the number of children exceeds some 
threshold like 40 (which is in my experience rare). 
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Figure 6: Example showing positioning heuristic. 


ftpd 


name-of-very-very-long- ftpd 
init <— inetd ftpd 
ftpd 


name-of-another-very- 





ftpd 

Figure 7: Crossover problems. This can be avoided 
by starting the vertex after inetd further to the 
right. Also note that following these lines would 
be easier if the difference in slopes is increased. 


ftpd 


name-of -very-very-long-program ftpd 
init < inetd ftpd 
ftpd 


name¢-of-another-very-long-program 
ftpd 
Figure 8: Crossover problems addressed by moving 
the drawing point further out. Increased sloped of 
lines may also increase readability. Compare with 
Figure 7. 


The sorting to arrange things into levels by par- 
ent is O(npe*) for » nodes, p non-leaf nodes, and 
where c is the maximum number of children a parent 
has. This is a rough analysis and looks unpromising, 
but in practice is probably pretty good. I suppose a 
radix sort could be done on the levels. Then we’d have 
O(n + pc’). 

Still someone might want to time various algo- 
rithms. The program is not linear. Faster tree-layout 
and arranging algorithms would be helpful in handling 
larger trees. 


Other Niceties of xps 


xps does a sort by user id and by process id 
within that. This tends to group related processes 
together and the process id arranges things by age. It 
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has been suggested that if it 1s desirable to sort pre- 
cisely by age, that should be done instead of sorting by 
process id. 


xps has the ability to filter out processes by user 
id or userid — regular expression. Each user is assigned 
a different color. Reducing the amount of display has a 
two-fold benefit: it not only unclutters the display but 
it also reduces the amount of time needed for display. 


Finally, xps has the ability to point and kill. (The 
interface hasn’t been developed to the point of arcade 
games, but still it might give the systems administrator 
a heightened feeling of power; in contrast to mass kill 
programs like skill, it can be satisfying to see the pro- 
cess die before your very eyes.) 


A screenshot from the program 1s given in Figure 
9: 


Future? 


Just about everything could be improved. The 
distribution contains an extensive list of things to do. 
Above were some suggestions for how tree layout 
might be improved. 


The thing I would most like to see is xps ported 
to the KDE qt and the GNOME gtk+ and glib libraries. 
Any volunteers? 


Display is done by drawing on a canvas. This 1s 
primitive. Process names are not widget labels, just X 
strings drawn on a canvas. Figuring out the process 
under the mouse and showing which process is 
selected is a bit low-level and not tool-like. Currently, 
a horizontal and a vertical linear search 1s done to find 
the process id under the mouse. Binary search might 
be faster. 


Alternatively, if a full-fledged widget for the 
nodes of a tree were used, then X (or an X-toolkit) 
would worry about when the widget is selected; pre- 
sumably it uses an algorithm at least as good as binary 
search. Making the selected node look, well, selected 
would then be done by the toolkit. 


It would be nice to benchmark the various toolkit 
approaches for efficiency. 


One might experiment to compare speed and 
intuitiveness of using a canned tree widget versus a 
hacked layout customized for this application. Person- 
ally, I prefer the tree layout, but the code is not toolkit 
idiomatic. 

Currently, the program contains coordinates of 
nodes before its position gets updated. However noth- 
ing else related to the history of the layout is saved, 
such as how long a node has been in the same posi- 
tion. It might be interesting to experiment with algo- 
rithms which make incremental improvements over 
time using say local heuristics. Analogous 1s the splay- 
tree algorithm which tends to make a tree balanced 
over time by making small localized changes as nodes 
are accessed. 
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Related work 


pstree by Werner Almesberger found on Linux 
distributions shows a static Manhattan-metric process 
tree relation using character-oriented graphics. 


Another program named pstree, by Lars Chris- 
tensen, does about the same thing and runs on other 
versions of Unix. Find it at ftp://ftp.thp.Uni-Duisburg. 
DE/pub/source/ . 


I. Herman, G. Melangon, M. S. Marshall “Graph 
Visualisation and Navigation in Information Visualisa- 
tion” can be consulted for a survey on graph visual- 
ization and navigation techniques, used in information 
visualization. For example it cites the classic paper on 
tree layout, E. M. Reingold and J. S. Tilford, “‘Tidier 
Drawing of Trees,’ IEEE Transactions on Software 
Engineering, SE-7(2), pp. 223-228, (1981). 


However a number of subsequent papers includ- 
ing those found in the survey given above question the 
tree heuristics used in this paper. For xps, I have not 
found this paper all that useful, although it is an inter- 
esting read. See the survey at http://www.cwi.nl/ 
InfoVisu/Survey/StarGraphVisulnInfoVis.html . 


Finally George MacDonald’s treeps program is 
similar. Find a home page at http://www.slip.net/gmd/ 
tps/treeps.htm. 


Availability 


See the project’s home page at http:/Avww. 
netwinder.org/rocky/xps-home. The program is avail- 
able from ftp://netwinder.org/users/r/rocky/xps.tar.gz . 
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Logging with SHARP 
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ABSTRACT 


System messages in a UNIX system are handled by syslog. The responsibilities of syslog are 
to filter and disperse program generated messages based on a priority code contained in each 
message. Filtering with priority codes 1s not sufficient to generate enough usable information for 
the system administrator. Utilities which do regular expression parsing of syslog messages 
typically do not run continuously and thus are limited by a lack of state in detecting potentially 
important patterns in syslog messages. 


SHARP (Syslog Heuristic Analysis and Response Program) improves the monitoring of 
systems by extending the existing syslog infrastructure with programmable modules. These 
modules use a library with a simple API to perform near real time analysis based on the messages 
they register to receive. System administrators can use SHARP to improve the services provided 
by their systems without the need for constant manual evaluation of message logs. The SHARP 
system and several modules were tested in a higher education production environment during the 
spring of 2000. Experience with SHARP indicates that it is stable, reliable, and improves the 
overall operation of a laboratory while not significantly increasing the workload on the system 


administrator. 


Syslog 


The “‘system logger”, or syslog, gives programs 
a standard interface to report interesting events to the 
administrator. These messages are read by a back- 
ground daemon and routed accordingly. The data 
which a program passes to syslog is called a message. 
A message consists of two parts: priority and textual 
data [9]. The priority of a message also contains two 
parts: an encoded facility and level. The facility of a 
message 1S a general category into which the message 
fits. The level of a message is a way for the program 
to rate the severity of the message, typically ranging 
from emerg to debug. The textual data of a syslog mes- 
sage 1s a string provided by the program that describes 
the event being logged. 


Programs use library calls to send a syslog mes- 
sage. The library allows the program to choose a facil- 
ity and level pair, as well as the text describing the 
message. The library will typically prepend informa- 
tion such as a timestamp, hostname, program name, 
and PID [7]. The library then delivers the message to 
the syslog daemon. 


The syslog daemon, syslogd, acts as the router for 
system messages. When it receives a message from a 
program, it in turn must decide what to do with the it. 
Most commonly this action involves writing the mes- 
sage to disk, but other potential actions include print- 
ing it to the system console, notifying online users, or 
forwarding the message to another system. syslogd 
makes these decisions based on a configuration file 
written by the system administrator. The rules in this 
configuration file are based entirely on the priority of 
the message. 
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Shortcomings of syslog 


The standard syslog daemon! lacks many impor- 
tant features. These features impact the the reliability 
of message delivery and the integrity of messages 
after delive ry. 


There 1s no standard structure for writing syslog 
messages. A cleverly written program could bypass 
the syslog library calls and write directly to the listen- 
ing syslogd socket. When syslogd reads this message, it 
will prepend default priority information and route the 
message according to these defaults [8]. While the 
lack of message structure 1s not critical for system 
operation, 1t does not encourage good programming 
form. 


When syslog messages are forwarded over a net- 
work the problem of time synchronization arises. 
There are techniques for network time synchroniza- 
tion, but even short skews of seconds are a problem. 
When syslogd receives a message from a remote host, 
it writes the message to disk as it was received and 
does not provide an additional timestamp. When ana- 
lyzing messages, this lack of consistent timestamping 
makes strict ordering impossible. 


Some versions of syslog have the ability to route 
messages based upon regular expression filtering. This 
allows greater discrimination and handling of mes- 
sages than 1s possible with priority filtering. Sophisti- 
cated classification and processing of messages 1s still 
difficult with regular expression filtering. Extending 
syslog in this manner violates the UNIX design 


'This refers specifically to the 4.4 BSD implementation 


[8]. 
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philosophy of simple tools doing one thing well. Extra 
processing must be performed with each message, 
which decreases message handling capacity. 


When the syslog system is compiled, the differ- 
ent facilities and levels are hard coded into the system. 
Most systems have eight different local facilities to 
give the programmer a set of alternative facilities. 
Some programs assume that a single local facility is 
available for exclusive use when this is not necessarily 
the case. With as few as eight local facilities available, 
there may be conflicts between programs. There is no 
way for the programmer to extend this limit without 
significantly altering the architecture. 


When syslogd writes a message to a file, all prior- 
ity information is lost. After the message has been 
written, syslogd drops the message, forever losing 
potentially important forensic data. The priority of the 
message could be useful in that it shows the state of 
the program that generated the message. While the 
text of a message is the most immediately helpful por- 
tion, it should not be treated as the only valuable 
piece. 


After the message has been written to a file, any 
person with write access to the file, authorized or not, 
may alter the contents of the file. While there is no 
way to stop any intruder with total access to a system, 
there are cryptographic protocols designed to detect 
alterations in log files [3]. These protocols write a 
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sister log file containing cryptographic information to 
verify the integrity of the messages. 


There is currently an IETF working group [2] 
that is attempting to create a new standard to solve 
many of these problems. 


Extensions to syslog 


Shortcomings in syslog and the need for 
improved message analysis have spawned several util- 
ity programs. These programs read syslog messages, 
perform analysis, and direct the results. These tools 
are used by administrators to automate analysis of 
messages. 


The simplest and most common type of message 
analysis is the grep-style filter. This type of program 
regularly parses syslog messages written to disk for 
predefined messages. In its simplest form, this pro- 
gram is a shell script spawned from cron that uses reg- 
ular expressions to grab messages from a log file [6]. 
These types of utilities have value, but only as post 
mortem tools. By the time the administrator has seen 
the output, a significant amount of time may have 
occurred where a potential problem may no longer be 
fixable. Imagine an administrator runs a regular 
expression filter nightly, only to find out the next 
moming that the company database has been filled for 
hours, rendering the system unusable. 


Another type of tool does the same style of pars- 
ing, but in real-time. These utilities, swatch [1] being 
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Figure 1: syslog and SHARP 
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the most prominent, read syslog messages in real-time, 
perform string based parsing, and direct the output. 
The limitation of this type of system is the ability to 
form complex routines based on messages. While you 
can configure this device to pass messages to an out- 
side program, state is not saved across invocations. 
Assume an administrator has a swatch filter that dials a 
pager each time the company database is filled. When 
a database fills, it usually complains repeatedly, not 
just with just a single message. With a swatch style fil- 
ter, the administrator will be paged as many times as 
the database error message occurs. 


SHARP 


All of these shortcomings with syslog point to 
the need for another tool that is able to perform real- 
time analysis and execute complex heuristics while 
retaining compatibility with the current style of system 
logging. The SHARP (Syslog Heuristic Analysis and 
Response Program) system attempts to address many 
of these architectural shortcomings. SHARP offers an 
interface for heuristic analysis programs to wisely uti- 
lize system logs. System administrators can decrease 
the burden of manually reviewing logs and improve 
the overall fluidity of the system by using SHARP. 


Architecture 


SHARP was designed for both simplicity and 
reliability. Simplicity is necessary to not obfuscate an 
already complex architecture of UNIX messaging. 
Reliability is necessary to guarantee the proper han- 
dling of messages and the overall stability of the sys- 
tem. A single mishandled message could lose informa- 
tion crucial to the administrator. 


The SHARP architecture consists of three parts, 
SHARP modules are registered processes that perform 
some sort of heuristic analysis. The modules use a 
simple library interface to connect to sharpd, the 
SHARP daemon. sharpd communicates directly with 
the syslog system and receives a copy of every system 
message from syslogd. The library does the work of 
connecting to sharpd and passing the received mes- 
sages back to the module. 


Using the library interface, resident SHARP 
modules can specify the priority of messages they 
would like to receive. When syslogd receives a mes- 
sage, it passes it to sharpd, which in turn passes the 
message to only the interested modules. The modules 
perform their heuristics and can take various actions 
such as logging, emailing, or notifying the administra- 
tor. 


When sharpd passes a message to a module, the 
message has been parsed and placed in a predefined 
structure to standardize analysis. This structure con- 
tains the time the message was received, the priority, 
the host that generated the message, and the text of the 
message. Communication with modules occurs over 
UNIX domain sockets. Using UNIX domain sockets 
provides further extensibility for replacement with 
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internet sockets. A SHARP system could forward 
messages to modules residing on separate systems for 
analysis. 


To report its findings, a SHARP module may 
choose to send a syslog message. If a module reaches 
a conclusion based upon previous messages, this new 
syslog message could report its findings at a higher 
level than the previous messages. This higher level 
message might be noticed by the administrator, while 
the previous messages of a lower level may have 
slipped past. Modules could use syslog messages as a 
rudimentary version of interprocess communication. 
Care must be taken by the module programmer not to 
generate messages that would travel in an infinite loop 
between the SHARP and the syslog architectures. 


Example Usage 


For example, a module could track user patterns 
and report anomalous behavior. There are multiple 
user entry points into a UNIX system: ssh, telnet, ftp, 
rlogin, and so forth. Since each point of access is 
equally valid, it would be insufficient to monitor only 
one. SHARP is a perfect system to correlate this login 
information. The SHARP module would indicate to 
sharpd that it only wishes to receive messages about 
logins. Over time, the module would build up a 
database of user patterns such as the time of login and 
the remote host from which the user is accessing the 
system. When the module receives information about 
a new login, it would report anomalous behavior based 
on a configurable degree of confidence. Suppose a 
user normally only logs in to the system from a 
machine local to the office and only during business 
hours. When the SHARP module sees the same user 
logging in from a system across the Intemet at 3am, it 
would report the strange behavior. 


A SHARP module could be written to handle the 
problem of an administrator being inundated with 
pages due to a flood of messages. This SHARP mod- 
ule would maintain a queue of incoming messages to 
be paged, and throttle similar messages. With this 
module, the administrator could fine tune the ability of 
programs to flood the notification path of urgent mes- 
sages. 


A Complete System: nsyslogd and SHARP 


As previously shown, the current syslog system 
is inadequate for a system such as SHARP to operate 
at full potential. A replacement for syslogd that does 
not have as many problems can be used. nsyslogd 
solves many of these problems by extending the func- 
tionality of the standard syslog system [5]. 


nsyslogd uses TCP instead of UDP for network 
message delivery. TCP, a connection oriented, reliable 
protocol, is much safer to use on long network paths. 
TCP is more difficult to spoof than UDP, but not 
impossible. nsyslogd can also use SSL over TCP, 
which gives network connections host authentication 
and data encryption. 
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nsyslogd uses a hash based algorithm [3] to guar- 
antee message integrity. For each file destination a sis- 
ter log is created containing chained hashes of the 
written logs. An external program is executed to check 
the integrity of the logs against the hash file. 


Other syslog replacements have similar features, 
but nsyslogd has one prevailing feature that is useful to 
the operation of SHARP: priority preservation. When 
nsyslogd writes a message to a destination, it includes 
the priority information. SHARP uses this feature to 
filter messages for delivery to interested modules only. 
Without this feature SHARP is forced to flood each 
message to every module, which in turn is expected to 
do its own parsing. 


sharpd communicates with nsyslogd by means of 
a UNIX domain socket. nsyslogd can be configured to 
write all system messages to a socket where sharpd 
receives them and passes them to interested modules. 


nsyslogd was written by Darren Reed and is 
freely available on the Internet. It has been ported to 
multiple UNIX platforms. It has proven to be a reli- 
able alternative to the standard syslog that, when 
paired with SHARP , becomes an even more powerful 
system. 


Modules 


Modules utilize the framework provided by 
SHARP. A module is defined as any program that uti- 
lizes the SHARP module interface and receives mes- 
sages from sharpd. Developers of modules include a 
single header file and link their module against the 
SHARP library. The interface is presented along with 
a simple example that illustrates proper usage. 


Module interface 


The interface to the module library consists of 
only three core functions and is designed to be as sim- 
ple and flexible as possible for the programmer. 


void sharp_filter (const char *fac, const char *lev) 
Register interest in messages of facility fac and 
level lev. A wildcard facility named all is 
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available that matches every facility. This func- 
tion may be called any number of times, but 
must be called before sharp_init(). 


int sharp_init (const char *name): 
Connect to sharpd and register with the corre- 
sponding name. 0 is returned on success, -1 on 
error. This function must be called before 
sharp_run(). 


void sharp_run (void (*callback)()); 
Begin receiving messages from sharpd. callback() 
is the function that handles a received message 
and must be of a particular prototype. sharp_run() 
is distinct from sharp_init() to allow for future 
developments in SHARP that perform run-time 
checks on modules when they connect to sharpd. 


void callback (sharp_msg m); ---begin:quotation---> 
The function defined by the programmer that 
performs analysis on the message passed as a 
parameter. When _ callback() returns, the 
sharp_run() function again waits for another mes- 
sage from sharpd. 


The data structure containing the message from 
sharpd is of the following format: 

typedef struct sharp_msg { 
fine-<t times /* sharpd timestamp */ 
chat Sprvlils- Ve priori ty 77 
ehar hostile t soripimatine ost" 7 
char text | + 7/ "message irextm./ 

} sharp_msg; 


Each string field is of a fixed length defined in 
the header file for security and robustness. Since this 
structure 1s just a copy of a message, the module is 
allowed to alter the contents of sharp_msg. 


The function atexit() is used to force execution of 
a SHARP library function that deregisters itself with 
sharpd. The programmer does not explicitly have to 
deregister the module. Figure 2 shows the state of a 
module as it connects to sharpd, receives messages, 
processes them, and finally exits. 
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Figure 2: SHARP module state diagram. 
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Example Module: mark 


To fully illustrate the simplicity and flexibility 
involved with authoring a SHARP module, an exam- 
ple is presented here to introduce the interface to 
potential programmers. This module is called mark 
because it reads syslog mark messages and reacts 
when a problem occurs. 


syslogd can be configured to generate a message 
of facility mark at a predefined interval. This message 
is a “heartbeat” to show that syslogd is still running. 
The mark module reads these messages and reacts if it 
does not receive one at a predefined interval. Listed 
below is the source code, edited for both clarity and 
brevity. See Listing 1. 


Note that this example is a skeleton and does 
require extra coding to be fully functional. The exam- 
ple only shows the code necessary to monitor syslogd 
on a single host. Adding support to the module for 
multiple hosts is straight forward. 


Also included with the SHARP system are utility 
functions that modules would often have to use, such 
as sharp_email(), shown in the above example. 


Current and Planned Modules 


The authors have implemented and tested several 
SHARP modules. It is hoped that the SHARP web site 


#Hinclude "sharp.h" 
#fdefine TIMEOUT 60 
Statice time. Jack: 


Void MatirGantarect char argv) 
{ 
Sharpofil ter (mark «aioe 
sharp init C'mark™) 
signal(SIGALRM, alarm); 
alarm(TIMEOUT) ; 
sharp sume (vold™) mark callback); 
} 


void mark_callback(sharp_msg m) { 
if(m.time-last > TIMEOUT) 
react 
else 
last=now; 


} 


void alarm. { 


/* Did not receive the mark message */ 


react(); 


} 


VOLd react () 4 


/* 
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comes to be a central point for module sharing in the 

spirit of the Open Source movement. The modules 

which we have implemented or anticipate implement- 

ing include: 

mark The mark module outlined above receives 
heartbeat messages from any number of syslog 
daemons on remote machines. If a message is 
not received in a configurable amount of time, 
the mark module assumes the remote host or 
network is down and informs the administrator. 

userpattem A module that gathers statistical data of 
the system use of users and reports anomalies. 
For instance, a particular user is known to only 
use the system during office hours and connects 
only from a particular host. When the module 
notices that the user is online at 3am from an 
off-site host, the module would report this 
anomaly. This shows the power of a heuristic, 
stateful approach in monitoring. 

useralert A module that alerts the administrator when 
a particular user connects to the system. This 
module can also be used to centralize login 
records for multiple machines. 

dailyfilter This module emulates the functionality of 
syslog utility programs. It filters out all syslog 
messages based upon regular expressions and 
email the output. 


/* threshold seconds */ 


* Possible reactionary methods include sending an email, 
* dialing a pager, or sending a syslog message. 


ei 


sharp_email("admin@localhost", "syslogd died") ; 


Listing 1: mark module. 
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problemalert Critical errors repeated more than a 
threshold number of times over a set period 
escalate notification (ie paging the sysadmin). 
Works well with the “‘all’’ wildcard. 

firewall This module receives firewall logs and ana- 
lyzes trends and anomalies. 


Experience using SHARP 


The Computer Science and Information Systems 
department at Grand Valley State University maintains 
a network of machines known as the Experimental 
Operating Systems Laboratory, or EOS lab. The EOS 
lab consists of 28 Linux workstations and one central 
file and authentication server. Because of its classical 
network architecture, this environment was optimal 
for testing a working implementation of SHARP. Due 
to the enormous amount of system messages, they are 
generally ignored by the EOS administrators and only 
used as an autopsy instrument. 


Results 


It was determined each workstation generates an 
average of 250 system messages per day. The main 
server for the lab generates an average of 36,000 sys- 
log messages per day. The total number of system 
messages generated in the EOS lab averages just 
above 42,000 per day. If this rate were normally dis- 
tributed over the course of 24 hours, a message would 
appear every 2 seconds, way beyond the sensory 
capacities of any human to manually monitor. These 
statistics are highly site and service dependent, but 
should offer a general idea as to message rates. 


The SHARP system has worked smoothly on this 
network. The concept of a centralized logging system 
seems to be the best operating environment for 
SHARP. Individual host installations of SHARP do 
not scale well when dealing with large networks. It is 
much easier to configure a new system to forward all 
syslog messages to a central logging system than to 
install SHARP and the various corresponding mod- 
ules. 


The EOS lab is a very small network in compari- 
son to larger installations. SHARP is expected to scale 
well to these much larger systems. Due to its nature, 
the architecture is well suited for scalability. The cen- 
tralized logging host could forward messages to 
instances of SHARP on different systems. Each sys- 
tem could have a unique set of modules. The socket 
architecture built into SHARP allows seamless 
changes between a local ==>[ignored: sc]}<== UNIX- 
domain socket and a network socket. SHARP would 
thus easily support applications such as distributed 
anomaly detection. 


Future Work 


With the experiences gained from using SHARP, 
the authors have noted a few features that would both 
aid module developers and improve overall perfor- 
mance of the system. 
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e Patches for other implementations of syslogd to 
pass priority information with messages. 
SHARP users will not be forced to use nsyslogd. 

e A global configuration file, such as /etc/ 
sharp.conf, would define variables that modules 
could use. For instance, a module would call 
sharp_get_var(ADMIN) to get the email address of 
the administrator defined in the configuration 
file. This way potentially dynamic information 
would not have to be hardcoded into modules. 

e Support for other languages besides C that have 
better text processing capabilities. Ideally a Perl 
module will be the first to be implemented. 

e Support threadSHARP library calls. 


Conclusion 


The traditional UNIX message facility is inade- 
quate for the active monitoring and response required 
of system administrators of even moderately large net- 
works. Working with the existing syslog infrastruc- 
ture, SHARP can improve the quality of monitoring, 
while at the same time reducing the burden on the sys- 
tem administrator. 


Our goals for the design of SHARP have been 
met. SHARP is compatible with syslog, and hence 
requires no changes to future or existing message gen- 
erating programs. SHARP ’s architecture is simple and 
clean, promoting extensibility and sharing of modules. 
The relatively small (approximately 2000 lines of 
code) implementation of SHARP 1s indicative of its 
simple and clean design. 


SHARP’s modules allow for continuous monitor- 
ing and the maintenance of state, which in turn sup- 
port the implementation of monitoring heuristics not 
possible with other extensions to syslog. The modules 
we have implemented work as expected and have 
proven their utility in a production environment. 


We have tested SHARP in a production environ- 
ment of a network of Linux workstations and a single 
file/authentication server. Our testing has shown 
SHARP to improve the monitoring of a network of 
hosts by supporting a pro-active form of monitoring 
by the system administrator. 


Availability 


SHARP and a few packaged modules are avail- 
able in source code form from the official web site: 
http:/www.csis gvsu.edu/sharp. This code is available 
under the BSD license [4]. Both the SHARP daemon 
and the modules have been compiled and tested on 
multiple platforms. 
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Peep (The Network Auralizer): 
Monitoring Your Network With Sound 


Michael Gilfix & Prof. Alva Couch — Tufts University 


ABSTRACT 


Activities in complex networks are often both too important to ignore and too tedious to 
watch. We created a network monitoring system, Peep, that replaces visual monitoring with a 
sonic ‘ecology’ of natural sounds, where each kind of sound represents a specific kind of network 
event. This system combines network state information from multiple data sources, by mixing 
audio signals into a single audio stream in real time. Using Peep, one can easily detect common 
network problems such as high load, excessive traffic, and email spam, by comparing sounds 
being played with those of a normally functioning network. This allows the system administrator 
to concentrate on more important things while monitoring the network via peripheral hearing. 


This work was supported in part by a USENIX student software project grant. 


Introduction 


Are your systems and network functioning cor- 
rectly? Can you be sure at this moment? Every admin- 
istrator has some need to be able to answer these or 
similar questions on an ongoing basis. 


Current approaches to live monitoring of net- 
work behavior (such as Swatch [10], mon [4], and 
their many relatives) can send email or page responsi- 
ble people when things seem to go wrong. These tools 
are both visual and intrusive; operators must either be 
interrupted by alerts or periodically suspend other 
work to check on network status. Furthermore, these 
approaches are highly problem-centered and provide 
mainly negative reinforcement; the monitor notifies an 
operator only when problems occur. It does not, as a 
rule, regularly inform one when things are going well. 


We created a tool Peep that represents the opera- 
tional state of a system or network with a sonic envi- 
ronment. The flavor, texture, and frequency of sounds 
played are used to represent both proper and improper 
network performances, while the ‘feel’ of the sounds 
provides the listener with an approximation of net- 
work state. This environment plays in the background 
while the operator continues other tasks. Without 
looking anywhere and without interrupting other 
pressing activities, the operator can hear peripherally 
whether action is required. 


Auralization 


The idea of auralizing network behavior by play- 
ing network sounds is not new. Joan Francioni and 
Mark Brown [3, 5] represented parallel computer per- 
formance using a synthesizer driven by a MIDI inter- 
face. The strength of this approach, however, was also 
its main limitation. For music to remain pleasant, one 
must limit one’s representations to a limited number of 
relatively pleasing harmonic combinations. This 
greatly limits what one can represent with this tech- 
nique. Earcons [2] are the sonic equivalent of icons; 
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sounds that are naturally associated with particular 
events. For example, most people associate a car hom 
with impatience or alert and a doorbell with someone 
entering a house. 


Both of these approaches define the meanings of 
specific sounds or particular combinations in isolation. 
Combining sounds is difficult unless they are conso- 
nant either musically or environmentally, that is, that 
the sounds naturally occur together and ‘sound right’ 
in combination. Natural sounds have an advantage 
over music; they sound normal and pleasing in almost 
any combination similar to that of nature. For exam- 
ple, birds and frogs in wetlands can sing with virtually 
no coordination, and the result is still pleasing. 


The Psychology of Audio Notification 


What makes Peep possible 1s that events in net- 
works ‘have easily recognized natural sound counter- 
parts. Moreover, numerous natural sounds can be 
played in combination while the result stays pleasing 
to the ear. If each sound represents some part of net- 
work function, and all are played together, the result is 
a sonic ecology in which the current state of the net- 
work can be determined moment by moment. 


Peep exploits human instinct: our ability to 
notice a deviation from the norm with little effort, to 
determine what sounds right, and to discern singular 
important sounds from a collection of many sounds. 
We do these tasks with little or no conscious effort. 
Since computer interfaces mainly require the visual 
senses (and some motor skills), the audio senses are 
left available to perform this unconscious processing. 


Furthermore, Peep takes advantage of our ability 
to do abstract processing. Instead of attempting the 
difficult and sensitive problem of determining when a 
network crisis has occurred or 1s about to occur, Peep 
provides contextual, continuous sound information 
and leaves interpretation to the listener. Decisions are 
based not only on the quantitative measure of things, 
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but the relative amount and absence of things. A musi- 
cian friend has often expressed to me his philosophy: 
‘‘Anybody can play drums, but the great drummer 
concentrates as much on the feel of the notes as on the 
space, or absence of sound, between them.” Similarly, 
information that is lacking from Peep’s sound 
ambiance is just as important as the amount of infor- 
mation conferred and the relative magnitude 1s left to 
the judgement of the listener. 


Representational Techniques 


Sound representation in Peep is divided into 
three basic categories: Events in networks are things 
that occur once, naturally represented by a single peep 
or chirp. Network states represent ongoing events by 
changing the type, volume, or stereo position of an 
ongoing background sound while heartbeats represent 
the existence or frequency of occurrence of an ongo- 
ing network state by playing a sound at varying inter- 
vals, such as by changing the frequency of cricket 
chirps. 


Peep represents discrete events by playing a sin- 
gle natural sound every time the event occurs, such as 
a bird chirp or a woodpecker’s peck. The sounds we 
chose are short and staccato in nature and easily dis- 
tinguishable by the listener. Additionally, we noted 
that certain events tend to occur together and found it 
convenient to assign them complementary sounds. 
While monitoring incoming and outgoing email on our 
network, we noticed that the two events were often 
grouped together, since both types of email were usu- 
ally transferred in a single session between mail 
servers. To better represent this coupling between 
Incoming and outgoing email events and make the 
representation sound more natural, we used the sounds 
of two conversing birds. Thus, a flood of incoming 
and outgoing email sounds like a sequence of call and 
response, making the sound ‘imagery’ both more 
faithful to our network’s behavior, as well as more 
pleasing to the ear. 


State sounds correspond to measurements or 
weights describing the magnitude of something, such 
as the load average or the number of users on a given 
machine. Unlike events, which are only played when 
Peep is notified of them, Peep plays state information 
constantly and need only be signaled when state 
sounds should change. Peep represents a state with a 
continuous stream of background sounds, like a water- 
fall or wind. Each state is internally identified as a 
single number measurement, scaled to vary from 
extremely quiet to loud and obnoxious. Background 
sounds should be soothing while the network is func- 
tioning normally. However, when the administrator 1s 
annoyed, he will know that action 1s required. 


Heartbeats are sounds that occur at constant 
intervals, analogous to crickets chirping at night. A 
common folk tale is that one can tell the temperature 
from the frequency of cricket chirps; likewise we can 
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represent network load as a similar function. Intermit- 
tent chirps might mean low load, while a chorus might 
mean high load. Heartbeats can also report results of 
an intermittent check (or ping) to see if a given 
machine, device, or server is functioning properly. 


Humans are very apt at recognizing when contin- 
ual background sounds change, making problem 
detection swift and simple. If your email server dies, 
chances are that you will not receive any email warn- 
ing of the problem. But the crickets will have stopped 
chirping. The heartbeats provide an effective method 
for monitoring the functionality of your network and 
being alerted of a problem when all else fails, through 
the absence of sound. Likewise, the administrator need 
not fear about monitoring his Peep server; if it dies, he 
will be immersed in sudden silence! 


Sound representation depends very much on per- 
sonal taste. Peep aims to provide users with a choice 
of themes such as wetlands (the current theme avail- 
able) or jungle. Within a theme, sounds are classified 
according to the network events they most appropri- 
ately express. Although the two chorusing birds were 
used to represent incoming and outgoing mail in the 
previous example, the two bird sounds could have 
been used for any type of coupled event behavior. 
These classifications help the user make decisions on 
what sounds to use from his collection of favorites. 


We also recognize that distinguishing sounds can 
be difficult if, for example, several similar bird sounds 
are used in a single theme. As the theme repository 
provided with Peep expands, we hope it will address a 
wide range of network situations and personal tastes. 


Scalability and Flexibility 


The Peep architecture was designed to be versa- 
tile and scalable. The architecture is based upon a pro- 
ducer/consumer relationship between distributed mon- 
itoring processes that watch the network and servers 
that actually play sounds. Producers alert consumers 
to events and state changes via short UDP messages, 
as shown in Figure 1. 


This architecture allows the receipt of status 
reports from any number of devices or nodes. Produc- 
ers (the monitors in Figure 1) monitor network behav- 
lor and report events and states while consumers take 
their input from the producers and play the appropriate 
sounds. Producers can be pointed at several sound 
generators simultaneously, e.g., a lab full of Linux 
workstations, for a truly immersive experience! 


Producers are executed as daemons on machines 
with access to information sources. This eliminates the 
need to send copious amounts of sensitive log or 
machine information across the network to a central- 
ized monitoring server. The packets sent to the con- 
sumer contain only sound representation information 
and would be of little use to a snooper without access 
to the Peep configuration file. 
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The Peep system was designed to take advantage 
of existing system administration tools. Server and 
client configuration information is stored in the same 
configuration file. This allows centralized control of 
Peep via simple file distribution via NFS or other 
widely accepted mechanisms such as CFEngine [6, 7, 
8] and rdist [9]. 


Clients provided with the Peep distribution are 
‘lightweight’ Perl scripts. Each client functions strictly 
within one problem domain: it addresses its original 
intended purpose and no more. This keeps client code 
simple, easy to debug, and easy to customize. 


We also wanted clients to run in the background 
and utilize as little resources as possible. Our log 
probing client, LogParser, watches log files and uses 
regular expressions to determine when particular 
events have occurred. Because of the way regular 
expressions are mapped in memory, scanning a single 
log for many different text patterns can become mem- 
ory intensive. Instead, we designed LogParser to dis- 
tribute monitoring overhead. Multiple instances of 
LogParser can run on separate feeds around the net- 
work, each instance searching for only a few textual 
patterns in the local system logs. This allows the sys- 
tem administrator to take advantage of the distributed 
computing power of his network, rather than waste 
what 1s often an abundance of idle resources in the 
hands of naive users. Peep aims to provide administra- 
tors with several means of implementing monitoring. 
Administrators still have the option of directing all log 
entries to a single machine should they so desire, at 
the cost of increased network bandwidth. Furthernore, 
the distributed method can be combined with the 
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single-machine method with no effort on the adminis- 
trator’s part. 


Expanding the capabilities of Peep to fit your 
own needs is simple. Perl libraries handle all the low- 
level details, so writing scripts for event, state, and 
heartbeat-driven feeds can be quick and painless. Log- 
Parser can also be easily configured to scan a log for 
new events via additional regular expressions. 


The Peep Protocol 


Peep was designed to allow centralized manage- 
ment of its distributed architecture. The Peep protocol 
uses auto-discovery to dynamically bind clients and 
servers together upon startup. Peep configuration also 
uses a class mechanism to define groups of clients that 
should all report data to the same servers. 


Peep was originally designed to use TCP for 
communication between clients and servers but com- 
munication over UDP proved much more efficient and 
effective. The main strength of TCP is its reliability. 
However, this reliability comes at the cost of greater 
bandwidth usage. Extra packets must be sent to 
ensure that transmissions were received correctly and 
in the proper order. Peep does not require packets to 
be ordered in any way — nor for packet transmissions 
to be reliable — since the representation of the state of 
the network ts an approximation rather than a precise 
depiction. In any case, the human ear has no way of 
distinguishing the exact order of events when events 
rapidly arrive at the Peep server; indeed, the resulting 
sounds seem simultaneous. 


The statelessness of UDP provided another bene- 
fit: clients and servers can be stopped and restarted 


———————— 
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Heartbeat 
Monitor 
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Figure 1: The Peep architecture. 
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without affecting one another. We wanted users to be 
able to write their own clients with minimal hassle. 
Avoiding connection management keeps clients sim- 
ple and allows one to readily write Peep clients with- 
out making use of the included Perl libraries. 

One drawback to using UDP 1s that clients have 
difficulty determining when servers crash. If this prob- 
lem is not addressed, a client will continue to provide 
data to a non-existent server forever. Peep deals with 
this problem by combining a leasing mechanism with 
auto-discovery. This combination provides safe, 
dynamic, real-time bindings between clients and 
servers. 


Peep’s auto-discovery mechanism uses a 
domain-class concept to maintain bindings between 
clients and their respective servers. When a server in1- 
tializes, it broadcasts its existence to the subnets asso- 
ciated with its classes and announces the classes of 
which it is a part. The clients that are members of 
those classes register themselves with the server and 
begin sending it packets. Conversely, should a client 
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start up and broadcast its existence, the servers associ- 
ated with its class will tell it to begin sending. A 
broadcast only occurs once during the initialization of 
each client or server, after which a list of hosts 1s 
maintained on both sides and communications are 
direct. Both clients and servers can belong to multiple 
classes at the same time and clients can communicate 
with many servers concurrently. 


Leasing is used to ensure that clients do not 
waste network bandwidth and system resources send- 
ing packets to servers that are no longer listening. The 
server sends a lease time to the client during auto-dis- 
covery. Just before the lease expires, the server tells 
the client to renew the lease. The client responds by 
telling the server that it 1s still alive and still needs to 
know about lease information. If the client has not 
heard from a server after the lease time has expired, it 
will no longer send packets to that server. Similarly, if 
a server does not receive lease acknowledgement from 
a client, it will no longer attempt to renew its lease 
with that client. 


Sleeping client 
| daemon 
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Figure 2: A server initialization. 
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Figure 3: A client initialization. 
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The auto-discovery and lease mechanisms 
greatly ease the burden on the system administrator. 
The system administrator can then use a file distribu- 
tion mechanism, like CFEngine, to add client and 
server daemons to a machine’s background processes. 
Clients will sleep until a server becomes available, and 
will send packets only while that server stays avail- 
able. 


Alternatively, system administrators may decide 
to dedicate a machine to run Peep software and want 
all clients to execute on a single machine. In this situa- 
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After compilation, the next step is to tell Peep 
which sounds to associate with which events, the 
classes to which your clients and servers belong, and 
your client configurations. A simple Peep configura- 
tion file is shown 1n Figure 4. 


class myclass 
Droadcast: 1307564. 23:-755-2000 
server swami:2001 

end myclass 


client LogParser 
class myclass 


tion, broadcasting becomes totally unnecessary and port 2000 

inefficient. Instead, the user can disable the auto-dis- config 

covery mechanism. Clients will then become dumb #Name |OptLetter |Location|Priority|RegX 

clients, continually processing and sending event out-mail O 1 “sendmail.*:.*from" 
Ihnc=ma isk i 255-20*. “Sendmatel es = to" 


information to a server throughout the course of their 
lifetimes. Peep also provides the user the choice of 
mixing and matching, applying distributed and cen- 
tralized configurations where they make sense. 


In terms of robustness, the Peep protocol has ver- 
sion identification, room for future expansion, and 
type identification. Upgrades should allow older 
clients to work with newer servers and vice versa. 
Communications are done using one-byte quantities to 
represent attributes, and strings for anything more 
complex. This allows us to avoid any external data 
representation issues, making the protocol more 
portable. 


Details of this protocol are hidden inside a Perl 
client interface library provided with Peep. The Peep 
library demands little expertise. To create a client 
with all of the library’s benefits, programmers need 
only initialize the library with their application name 
and tell the library what information to send. Initializ- 
ing the library parses the Peep master configuration 
file, so programmers need not do it themselves. This 
allows client design to be as simple or as complicated 
as the user desires. We hope that the simplicity of 
writing clients with the Perl library will encourage 
users to write their own client applications and share 
their code with others. 


Configuring the Peep System 


How one configures Peep is very much depen- 
dent on whether you choose to use single or multiple 
nodes. The generalized Peep installation is a four-step 
process: downloading the source and a sound package, 
compiling the server, editing the configuration file, 
and deploying clients. 


The Peep server package uses the gnu autoconf 
package to make configuration and compilation easy. 
Support for tcp wrappers [11] can be added as an 
option. Peep comes with two generic sound modules. 
One handles generic /dev/audio support while the 
other takes advantage of ALSA [1] on Linux systems. 
The configure package will default to ALSA drivers 
Over generic support, 1f present. Special support for 
the Sun audio jack 1s also provided. 
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end config 
end client LogParser 


events 
#Event Type|Path|]# sounds to load 
out-mail /path/sounds/peepla.* 1 
ine-mail /path/sounds/peep24a.* 1 
end events 


states 
#Event Type|Path|# sounds| Fade time 
loadave /path/sounds/water.* 5: (0.3 
end states 


Figure 4: An example peep.conf. 


Class definitions consist of two lines: one speci- 
fying broadcast zones and another specifying which 
servers are part of that class. Several broadcast zones 
and servers can be specified. Clients and servers can 
be part of several classes and will broadcast all the 
classes to which they belong during initialization. 
Putting multiple servers in a class (or making a client 
a member of multiple classes) is an easy way to have a 
single client dump data to multiple servers. 


The ‘events’ and ‘states’ sections tell Peep 
servers to associate a name with a group of sounds. 
Filename descriptions in the Peep configuration file 
have a trailing asterisk extension followed by the 
number of sounds to load. Peep expands each asterisk 
into a two-dig it number and loads, in ascending order, 
the number of sounds specified. All of the sound files 
loaded for a single entry then correspond to a single 
event. Every time that event occurs, the server will 
randomly play one of the associated sounds. This ran- 
domness makes the sound ambiance more natural. 
Heartbeats are created from streams of normal events 
from a client at suitable intervals. For state sounds, the 
server randomly strings together sound segments to 
create a non-repeating, random-sounding background 
ambiance. To keep transitions between sound seg- 
ments sounding natural, the user can specify a linear 
fade time between segments. 


The final step is to configure and deploy some of 
the clients provided with Peep. Two of those are dis- 
cussed here: Peck and LogParser. 
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Peck 


Peck is a command-line utility provided with 
Peep. It allows the user to tell a server to play (and 
how to play) a given sound. Peck is an example of a 
dumb client and bypasses the auto-discovery and leas- 
ing mechanisms. Event and state attributes are speci- 
fied on the command-line and delivered directly to the 
server. Some command-line options apply to event 
sounds and others to background sounds, but the user 
need only remember a small number of options to get 
the Peep server to play some interesting things. Peck 
can be called with appropriate arguments from a shell 
script if a user does not wish to use a client library. 
Ideally, one should only utilize Peck to talk to servers 
on the same physical machine, or to report very infre- 
quent events since Peck’s inability to use auto-discov- 
ery and leasing capabilities means that calling ap plica- 
tions will have no knowledge of the state of the 
receiving server. Peck is handy for a variety of simple 
tasks, including debugging installations, testing how 
things sound together, experimenting with Peep’s 
capabilities, and interfacing Peep with other monitor- 
ing systems (such as an existing Swatch or mon instal- 
lation). 


LogParser 


A simple log analyzer, similar to Swatch, is also 
provided with Peep. LogParser takes advantage of 
Peep’s auto-discovery and leasing mechanisms. It 1s 
also an efficient distributed tool. LogParser reads its 
entire configuration but only searches for and remem- 
bers textual patterns specified on the command-line. It 
was designed to have multiple instances run on several 
different machines, each scanning for different sets of 
textual patterns on each client machine. 


LogParser is flexible, easy to configure, and pro- 
vides a simple way to access Peep’s capabilities for 
representing events and states. It analyzes log mes- 
sages as they are added to the log file and scans them 
for regular expressions. LogParser uses simple config- 
uration syntax to generate command-line options and 
determine which sounds to associate with which par- 
ticular events. Several options follow: 

e The priority of the event ensures that no matter 
how many network events hit the Peep server, 
the most important ones will be played first and 
foremost. 

e The stereo location of the event, aside from 
pleasing the true audiophile, helps the user dis- 
tinguish and even locate an event. Sonic loca- 
tions can even be assigned to correspond to the 
actual locations of machines on the network. 
Future versions of Peep might include a visual 
sound location map to exploit this. 

e A regular expression that tells LogParser how 
to find the event in a log file. Users with expe- 
rience with Awk/Perl pattem matching will 
appreciate this feature while others may find 
writing these difficult. We feel this is the easi- 
est way to extend the capabilities of Peep with- 
out doing any sort of programming. 
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Directives in the LogParser configuration can be 
enabled or disabled via command-line options. Each 
line of the LogParser configuration corresponds to a 
user-specified single-letter option. In Figure 4, incom- 
ing and outgoing mail are mapped to command-line 
options “]”’ and “‘O’’, respectively. Thus, an invoca- 
tion of LogParser searching for incoming mail might 
look as follows: 


LogParser -events=l 
-logfile=/var/log/messages 


Should the user forget the options, a help option will 
conveniently generate a list of user-configured 
options. 


A single instance of LogParser can scan numer- 
ous logs simultaneously. It can send event streams to 
multiple servers automatically via the auto- discovery 
and domain-class mechanisms. These features provide 
the user with a myriad of options for structuring the 
architecture of Peep within a network. 


Peep Performance under Pressure 


To deal with copious amounts of incoming net- 
work data, Peep has a queuing and windowing system 
that handles large numbers of simultaneous events. 
This ensures that events are played in the order of 
receipt and in accordance with their particular priority. 
Peep will also discard events from its queue if too 
much time elapses between receipt and playtime, in 
order to keep events relevant. 


Peep plays sounds by mixing sources in soft- 
ware. Since having large numbers of simultaneous 
voices can become computationally expensive, the 
user can tweak Peep’s performance by changing the 
number of voices used when mixing sound. Less mix- 
ing voices tend to mean that the Peep’s queuing and 
windowing system gets more usage, but the two 
always strike a balance to keep events accurately posi- 
tioned in terms of time of occurrence. 


It is difficult to send events to a Peep server fast 
enough to fill a queue on a Pentium II 400 and during 
testing, this required the use of an infinite loop. If the 
Peep server does manage to become overloaded, it 
only falls behind time-wise, adding a delay between 
the real-time event and the playing of its counterpart. 
Peep will preserve the general order and users will still 
be able to diagnose problems based upon the relative 
frequency of events. The delay experienced only 
applies to events and heartbeats; state changes occur 
instantaneously. In a worst case scenario, should the 
queue manage to fill up while new events are still 
arriving, Peep will begin discarding the oldest events 
from the queue, attempting to give the best approxi- 
mation of network activity. 


A Brief Overview of Implementation 


The inner-workings of a Peep server are based 
upon the interactions between three execution threads 
as shown in Figure 5: the listener, the engine, and the 
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mixer. The listener handles all communications with 
the client, discovering clients via auto-discovery and 
keeping track of client leases. Upon receipt of event or 
state data, the listener thread places the information 
into a queue to be processed by the engine. The engine 
works closely in conjunction with the mixer to keep 
track of the priority of incoming and currently playing 
sounds. The engine also tries to find the best available 
mixing channel on which to play the incoming events 
and informs the mixer of the necessary parameters to 
properly represent the information. Should a suitable 
mixing channel not be found, the engine will place the 
events into a priority queue, ensuring that the mixer 
will play the most important events as soon as mixing 
channels free up. The mixer performs the processing 
necessary to produce Peep’s output. This process 
Involves scaling each sound’s volume, as well as fad- 
ing between state sounds. The mixer must also check 
the engine’s event queue and ensure that queued, older 
events have priority as soon as mixing channels free 


up. 
Critique 


From our perspective, the design of Peep is very 
robust and portable. We decided, however, that sup- 
port for generic audio hardware was more important 
than efficiency of memory and processor usage on the 
server side. Peep utilizes Linux ALSA and OSS 
drivers, as well as the Solaris /dev/audio interface, to 
avoid device incompatibilities. This is done at the 
expense of ignoring commonly available device- 
dependent hardware-based mixing in favor of mixing 
in software. Software mixing did afford us one advan- 
tage that hardware cannot guarantee: users will always 
get the benefit of sound processing incorporated into 
Peep regardless of the hardware. Future plans do 
include support for hardware-based mixing on a 
selected number of audio cards. 


An invisible limitation of Peep is that creating 
accurate natural venues of consonant sounds is both an 
art and very labor-intensive. Due to copyright limita- 
tions on existing natural sound collections, Prof. 
Couch has spent many hours with a Telinga parabolic 
nature microphone and Sony DAT or digital minidisc 
recorder in search of the perfect bird. Sounds we col- 
lected required significant post-processing, including 
high and low-pass filtering and noise reduction, before 
they were free of enough normal background noises to 
Serve as event sounds. Collecting state sounds proved 
even more difficult, with the sound of wind being the 
most difficult. The challenge was to collect ‘desirable 
noise’ without impurities such as car horns and air- 
plane engines. 


In spite of the excellent guidance on the record- 
ing of natural sounds that we obtained from the Cor- 
nell Ornithology website [13], the Stokes Field Guide 
to Bird Songs [14, 15], and the British Library 
National Sound Archive [12] we are not ornithologists 
and apologize in advance for any gross mislabeling of 
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sounds included with Peep! Nonetheless, we have 
made significant progress in providing a Wetlands 
venue, and are planning others in the future. 
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Figure 5: The Peep server’s internal structure. 





Configuring a Peep theme pleasingly can be non- 
trivial, especially when choosing which sounds should 
be associated with which events. The process of 
choosing sounds can often be a very lengthy. Since 
sounds chosen vary according to personal taste and the 
situation they are attempting to describe, we hope to 
provide several different preset configurations for our 
users after the tool has had more exposure. 


Peep is relatively young and prior to this publica- 
tion has received very little public usage. We hope we 
have anticipated and met the needs of a wide range of 
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network implementations. However, only public usage 
and time will tell. 


Future Work 


We want to see several other capabilities added 
to Peep servers to better represent network events. 
One idea is ‘log dithering’. Due to block buffering, 
many log files are updated in erratic bursts so that sev- 
eral events are written to the log file and reported by 
LogParser as simultaneous. A dither time would space 
out how the events are played so they have a truer rep- 
resentation. 


We also want to represent state sounds in a way 
that better models the way the human ear works. Since 
the ear hears amplitudes on an exponential scale (in 
dB), we want to scale state measurements exponen- 
tially so that they better approximate what the human 
ear considers truly loud. This still may not satisfy our 
vision of having a storm break loose when a machine 
is overloaded. 


We may also allow sounds to change in nature 
with volume. A small stream might become a river 
rapid when a state measurement, such as load average, 
Increases. State sounds might be represented by three 
or four different collections of sounds to achieve a 
‘thunderous’ effect. A final item on the server wish- 
list is pitch bending: the ability to play sounds at dif- 
ferent frequencies. Using this capability we could gen- 
erate birdcalls at different pitches and then combine 
them together to create the effect of a chorus of dis- 
tinct birds from a single sample. 


We would also like to add a GUI to ease the pro- 
cess of configuring sounds for Peep. Since we plan on 
having several different sound classifications, a sound 
browser would be a welcome addition. The interface 
would let the user play several sounds simultaneously 
so they could get a feel for how things would sound in 
various situations. This will most likely be the next 
major addition to the Peep software package. 


Lastly, we hope a few brave users will contribute 
homegrown scripts and configurations to the project 
so that we can establish an archive and ease the pro- 
cess of making a new installation. 


Conclusions 


This work began two years ago by trying to 
define what constitutes ‘normal’ behavior of a net- 
work and how to take action to rectify ‘abnormal’ 
behavior. This proved infeasible because normalcy 
depends as much upon policy decisions as upon many 
pre-existing conditions. These conditions exhibit com- 
plexities and intricacies that are difficult to depict via 
traditional methods. 

Our sound ecology depicts normalcy in a new 
way. Things are normal when Peep “sounds like it did 
yesterday,” regardless of the intricacy of the depic- 
tion. Our innate human abilities to detect these 
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differences are more acute than one may realize. 
When things sound different, we may not know why, 
but we can tell that something has changed. 


Traditional tools look for specific problems 
while Peep only tells the listener about potential prob- 
lems. In that respect, Peep will outlast traditional 
problem-detection tools because it portrays the general 
problem and no more. And unlike other tools, Peep is 
non-intrusive. One doesn’t need to pay much attention 
to Peep in order to benefit. We don’t want you to. We 
just want you to sit back, and listen. 


Availability 


The current revision of Peep is 0.3.0alpha and is 
is currently freely available from http://www.eecs. 
tufts.edu/peep/. A demo of Peep’s capabilities will 
also be provided on the website in .wav format so 
users can know what they’re getting into before they 
install it. 
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ABSTRACT 


Thresh is a simple SNMP [1] monitor, written in Scotty [2] (Tcl [3] with the Tnm 
extensions), which uses the UNIX file system hierarchy for configuration and data storage. Thresh 
compares SNMP variables to per-device thresholds or values, and issues notifications if any 
current SNMP variable values are unacceptable (or unexpected). Thresh can be used by itself, or 
as a complement to other network management and monitoring tools. Thresh can be thought of as 
fitting in between tools that generate immediate emergency alerts (such as Big Brother [4]) and 


trending and history tools (such as Cricket [5]). 


Introduction 


Virtually every computing system and network 
has some kind of monitoring mechanism in place 
these days, but in some cases the monitoring system 
consists only of users phoning and asking if there ts 
something wrong with the network. For those inter- 
ested in something a little more advanced or auto- 
matic, there are quite a few software packages avail- 
able that do some form of monitoring. They range 
from the very simple (ping tests, etc.) to the very com- 
plex (large commercial packages that map networks, 
configure devices, and make your lunch), with various 
alternatives in between. 


In the realm of “simple” monitoring software, 
packages can generally be divided into two types: 

e Alarmers — Software that monitors connections, 
services, and so on, and sends out an alarm 
(mail, pager, smoke signal) as soon as it detects 
a problem. Some examples of alarmers are Big 
Brother [4], nocol [6], and Spong [7]. 

e Trenders and Trackers — Software that keeps 
history or trend data for future analysis or 
review. The most obvious examples here are 
MRTG [8] and Cricket [5]. 


A couple of years ago it occurred to me that 
there was another class of monitoring that didn’t seem 
to be very well addressed — low to medium priority 
tracking of certain parameters and their values on 
computing systems and network devices. For example, 
you might want to track configuration changes, system 
or device reboots, or network interface status or 
change time. These are things that you may want to 
know, but which don’t necessarily indicate an immedi- 
ate problem and which may not be worth waking any- 
one up to investigate. 


Thresh was created to provide this kind of moni- 
toring. It tracks SNMP variables, compares them to 
threshold, pre-set, or last-observed values, and reports 
(typically via email) unexpected changes or out of 
range values. 


Why Use Thresh and Not Something Else 


I’ve become convinced that, ugly as it may 
sometimes seem, the Simple Network Management 
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Protocol (SNMP) [1] should be the basis for just about 
every monitoring system. Virtually every network 
connected device either comes with an SNMP agent or 
can run one with only a modest amount of effort. Most 
SNMP agents can provide a vast amount of (usually) 
useful information, and most agents for general pur- 
pose computers can be extended to provide just about 
any data that you might want to have. 


Some monitoring systems (such as Big Brother 
and Spong) rely, to a greater or lesser extent, on sepa- 
rate client agents, rinning on each system that needs 
to be monitored, with a system-specific reporting pro- 
tocol between the agent and the management station. 
This approach can be somewhat limiting (it’s hard to 
use on things like networking equipment for example), 
can result in some duplication of services (if you need 
an SNMP agent for other purposes), and limits both 
what you can do (they're often not extensible), and 
where (as your firewall may not be able to pass the 
particular protocol implemented by the software). 
Thresh avoids these kinds of problems by using only 
SNMP for communication. ' 


It’s often useful to be able to track certain SNMP 
variables, but you don’t always need to know about 
changes immediately — 1t’s often good enough to hear 
about them the next time you read your mail. For 
example, the system.sysUpTime.0 SNMP _ variable 
gets reset every time a system or device gets rebooted 
— if you get notified every time that variable resets, 
you'll know if you’ve got a device reliability problem 
(or an extension cord that people keep tripping over). 
Similarly, you can generate disk capacity threshold 
wamings, network bandwidth wamings, network 
interface up/down notices, and so on. 


With many monitoring systems, this kind of low- 
priority information can be hard to generate. Many of 
the most common freely available packages tend to 
have only a small number of notification or alert 
mechanisms, and are built with the idea that you’re 


'SNMP suffers in some situations from being a UDP based 
protoco], but its advantages more than make up for its few 
disadvantages. 
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monitoring vital services and networks — sometimes 
everything is assumed to be an emergency. 


Thresh was created to address this kind of 
medium-level monitoring need. 


There are a number of commercial monitoring 
systems available. Some, such as Spectrum [9], HP 
OpenView [10], and NetCool [I!], are widely 
deployed and very well respected, and most of them 
can do (or can be made to do) most or all of what 
thresh does. However, the commercial packages tend 
to require a much larger monitoring infrastrucutre, and 
a much larger committment of time and money to 
implement, operate, and maintain. For small to 
medium sized sites, smai!l, simple monitoring tools, 
like thresh, are often the best choice. 


Implementation 


Thresh was implemented in Tcl [3], using the 
Tnm network management extensions provided by the 
Scotty/Tkined [2] software. The Tnm extensions are a 
toolkit of Tcl procedures that make it easy to perform 
SNMP operations. 


Tcl was chosen because it is well suited to this 
kind of task, and the Tnm extensions provided just the 
right functionality. At the time thresh was first con- 
templated, the SNMP modules for Perl were relatively 
primitive when compared to Tcl and Tnm. 


Thresh has benefitted from the use of a scripting 
language, and the string and array manipulation rou- 
tines provided by Tcl. Tel allows the use of an interac- 
tive “shell’’ for testing and development. Being a tra- 
ditionalist, I tended to develop incrementally, using 
the well-known edit, run, repeat cycle, rather than a 
more “‘modern” approach to program development. 
Thresh is currently about 700 lines of Tcl. 


In retrospect, I’m still glad to have chosen Tcl in 
preference to Perl, C, awk, or Visual Basic. 


Simplicity 


One of the design and implementation goals for 
thresh was simplicity. That goal has been addressed in 
the following ways: 

e One main program, thresh, and a very small 
number (currently one) of simple utility pro- 
grams. 

e Implemented in a well-designed scripting lan- 
guage (Tcl [3]), using a well-known and effec- 
tive set of library routines (Scotty). 

e Simple to run — requires no additionai daemons 
(just a crontab entry), and no addtional software 
is required on remote systems?. 

e Requires no special userids or groups. (ft would 
probably be useful to have a separate userid to 
run thresh, and perhaps a userid or group to 


2Other than an SNMP agent which you should already 
have installed and configured anyway, and which likely 
came with your operating system. 
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own the config files, but that decision is left up 
to those installing and using the sottware.) 

e Configuration is relatively straightforward, and 
involves only two simple file formats. The hier- 
archical configuration structure makes imple- 
menting default settings easy and flexible. 


Data Direction and Configuration 


Thresh’s configuration is “data-directed”’? — it is 
configured using a hierarchy of directories and config- 
uration files that is intended to reflect organizational 
structure and DNS naming conventions. 


The default configuration assumption 1s that each 
directory in the configuration hierarchy represents an 
element of the DNS name of the devices being moni- 
tored. For example, the sub-directory named 
com/whizbang/admin/printerl would usually contain 
the thresh configuration for the device with the DNS 
name printerl.admin.whizbang.com. The name con- 
figuration directive makes it easy to override this 
default behaviour, by setting the DNS domain or node 
name associated with a particular directory. 


Configuration Variables 


Each directory may contain a DEFAULTS file, 
which sets the various configuration variables (such as 
name, notifier, delay, community, etc.) which control 
thresh’s behaviour. Thresh configuration variable 
names are case sensitive, and all include only lower 
case letters. Settings in a DEFAULTS file are in effect 
for that node and those lower in the hierarchy, unless 
overridden by a lower DEFAULTS file.* Thresh config- 
uration variables can also be set on the command line, 
in which case they override any other settings for the 
given variables. Figure 1 shows a sample DEFAULTS 
file. 


verbose = true 
name = mydomain.net 
community = hello 


mib = /usr/local/mibs/ascend.mib 
## big network, long timeout 
TIMeOwE. = 20 


notifier = threshmail jsellens 
sysilop -=—leoca ll arnto 


Figure 1: A sample DEFAULTS file. 


Thresh’s configuration variables include: 

e walkonly — Walk the data hierarchy, listing the 
nodes and variable references found, but not 
querying, reporting, or logging. 

e ignore — Defines a list of glob patterns of file and 
directory names to ignore. Setting 


ignore = * 


for example, ends up ignoring the data hierar- 
chy below a given point. 


3Or, at least, my definition of data-directed — I looked un- 
successfully for a more commonly accepted definition of the 
term. 

4One could claim that this is similar to the class inheritance 
rules in object-oriented programming languages. Or not. 
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notifier - The command line to run to send noti- 
fication messages — the message text is pro- 
vided as the standard input to the command. 
This can be set differently for different parts of 
a hierarchy, which makes it possible to assign 
responsibility for different nodes to different 
people or groups. 
e frequency — The interval (in minutes) before oth- 
erwise identical notifications may be sent. 
e describe — Whether or not to include a variable’s 
MIB description field in notification messages. 
Description fields often end up being fairly 
generic, but including the description im mes- 
sages can help make the notifications a little 
more self-documenting. 
syslog — A facllity.level pair for logging messages 
to syslog. 
The other configuration variables are described in the 
included threshvars(5) man page. 


SNMP Variables 


All other files in a directory are expected to con- 
tain a list of SNMP variables to monitor for that par- 
ticular device, with comparison indicators and 
expected or threshold values. Thresh currently lacks a 
file inclusion mechanism, but the use of symbolic 
links makes it easier to manage the configurations for 
multiple, similar devices. A sample configuration file 
is shown in Figure 2. 


The SNMP variable names used in a configura- 
tion file can be any string that Scotty will recognize as 
a particular MIB variable. They can be fully-qualitied 
names, such as iso.org.dod.intenet.mgmt.mib-2.sys- 
tem.sysUpTime.0, unique substings with common pre- 
fix elements removed, as in system.sysUpTime.0, or 
SNMP _ object’ identifiers (O1Bs), such as 
1.3.6.1.2.1.1.3.0. OIDs aren’t always the best choice, 
as they are typically somewhat more cryptic for the 
casual reader. 


The first letter on each configuration line, C, G, }, 
L, S, or V, indicates the comparison to be made: 
e C: Changeable — the variable’s value may 
change, but should be reported each time it 
changes. This is useful for semi-static data, or 


this is a comment 

system.sysDescr.0O 
system.sysContact.0 
system.sysUpTime.0O 
interfaces.ifTable.ifEntry.ifDescr.5 
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for monitoring things such as device interface 

status changes. 
e G: Greater than — the variable is reported if its 
current value is less than or equal to the speci- 
fied value. 
|: Increasing — the variable is reported if its cur- 
rent value is less than its previous value. This 1s 
handy for watching for reset times, such as the 
system.sysUpTime.0 variable resetting when a 
device (or agent) restarts. 
L: Less than — the variable is reported if its cur- 
rent value is greater than or equal to the speci- 
fied value. 
S: Static — the variable is reported if its current 
value is not exactly equal to the specified value. 
If no value is specified, then it is compared 
against the first-retrieved value of the variable. 
This 1s useful for monitoring things that should 
never (or almost never) change, such as sys- 
tem.sysName.0. 
V: Variable — the value can be anything, but it is 
queried and tracked to allow for later investiga- 
tion or review. 


The final field on some lines is the threshold 
value to compare against — a threshold value is 
required for G and L, optional for S, and not allowed 
for C, | and V. If a value for an S comparison is not 
provided in a configuration file, the first value for that 
SNMP variable retrieved from the device is saved and 
used as the “‘normal”’ value for the variable. 


The configuration mechanism has proven to be 
quite flexible and easy enough to deal with, though 
some form of file inclusion mechanism would make 
some configurations simpler to create and maintain. 


Notifications 


Thresh provides a flexible mechanism for notifi- 
cations. For each device described in the configuration 
hierarchy, if thresh determines that something needs to 
be reported, it formats a message and pipes it into 
whatever “notifier”? program has been specified by 
the configuration variables. Thresh also keeps a copy 
of the message for internal reference when it Is next 


90000 
3000 
8) 
175.0 
200 


Figure 2: A sample configuration file. 
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run. If thresh observes the same problems (i.e., an 
identical notification message) the next time it is run, 
it will only send another notification if the specified 
frequency time has passed. If it sees a different set of 
problems when next run, it will report the complete 
current set of problems, regardless of whether or not 
the normal delay time has passed. 


If the syslog variable is set, thresh will also gen- 
erate syslog messages for SNMP variables that are out 
of range, in addition to the normal notifications. Mes- 
sages formatted for syslog are deliberately terse, and 
include the node name, comparison type, SNMP vari- 
able, the normal or threshold value, and the current 
value. 


Status, History and Logging 


For each device in the hierarchy, thresh will cre- 
ate a configuration subdirectory named .thresh, as a 
convenient location to store the data it generates. 
Thresh maintains status, history and logging informa- 
tion, both for internal use and to make it possible to 
review past changes in state. It tracks the previous val- 
ues of the SNMP variables (in .thresh/status_*), the last 
notification message sent (in .thresh/last_complaint), the 
date and time of last contact with the device (in 
thresh/last_response), and a log of when variables were 
found to be outside their “normal” ranges (in 
thresh/log_*). 


The status and log files are named for the config- 
uration files that they are related to, with a prefix of 
status_ or log_ added to the configuration file’s name. 


Currently, there are no really interesting ways to 
access and use this data — you’re more or less stuck 
with using some paginator program to view the files. I 
should note that there is no built-in mechanism for log 
file rotation. 


Integration with Other Systems 


Thresh can be used effectively as a standalone, 
isolated monitoring tool, but it can also be integrated 
with other logging or reporting systems. Thresh’s core 
functionality is polling SNMP variables, comparing 
against pre-determined thresholds, and generating 
messages for distribution. Integration with other tools 
can be accomplished in two ways: 


Syslog 
Thresh can be configured to record out of range 
values to syslog, which provides an easy interface 
to any existing syslog watcher. 


Custom Notifiers 
By setting the notifier configuration variable, 
thresh’s alert messages can be trivially piped 
through arbitrary custom processes, that can 
record, mail, or dispatch as appropriate. 


The message format is fairly consistent, simple, 
and relatively easy to parse. Additionally, the inter- 
nal thresh code which actually generates the 
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messages would be easy to change if a specific 
output format was required.°® 


Scalability 


Thresh is not arbitrarily scalable to huge num- 
bers of devices and variables being monitored. 
Beyond a certain point, you will start to run into prob- 
lems such as: 

e Configuration complexity — a file and directory 
based configuration mechanism works fine up 
to a point, but beyond that you need configura- 
tion generators and a real database. 

No inherent parallelism — beyond some number 
of hosts and vanables, you won’t have enough 
time to poll everything you want to poll within 
the time interval you would prefer. You can 
deal with this to some extent by running multi- 
ple parallel instances of thresh, but that adds 
administrative complexity. 

The default email notification mechanism can 
quickly get out of hand, if things start breaking. 
It is possible to use a smarter notifier, or to send 
notifications to different users or aliases based 
on location in the hierarchy, but that can get 
complicated. It would be possible to use a noti- 
fier that inserts messages into your manage- 
ment console, but if you have one of those, you 
may also already have more advanced commer- 
cial monitoring software. 


Installation 


Installation of thresh is very straightforward. 
You need relatively current versions of Tcl and 
Scotty (at least 2.1.x) on your monitoring host. 
Set the “hash-bang” line at the top of the 
thresh script to point to your Scotty installation. 
Install the script and man pages in the “‘usual’”’ 
places. 

Set a cronjob to invoke thresh periodically, with 
a command line something like: 


thresh topdir=/path/to/data 
e And simply create a data hierarchy that 


describes the hosts, devices and variables that 
you wish to monitor. 


The last step is admittedly somewhat more compli- 
cated and time-consuming than the others, but that is 
pretty much unavoidable. The distribution will include 
some sample configurations. 


Enhancements 


There are a number of enhancements to thresh 
that should probably be made: 
e Thresh currently only supports SNMP V1 — it 
would be useful to enable all versions of SNMP 
supported by the Tnm extension. 


‘The msgformat configuration variable provides a rudimen- 
tary and somewhat unsophisticated mechanism for cus- 
tomizing the notification messages, 
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e Syslog logging is implemented through the 
external Jogger(1) command — it should be re- 
implemented using an internal library call. 

e There should be some form of “include file” 
mechanism for configuration files, to reduce 
the need for the use of symbolic links to imple- 
ment common configuration settings. 


Lessons and Problems 


Thresh seems to serve to illustrate a few useful 
lessons: 

e Sometimes a variety of tools are needed to 
implement a complete monitoring “‘solution’’. 

e Simple tools are often quite useful. 

e ‘‘Data-directed”’ design can be quite effective. 

e SNMP can provide all sorts of useful informa- 
tion that can be difficult or impossible to obtain 
using other mechanisms. 

e Tcl and the Tnm extensions are very useful 
tools. 


Some of these also illustrate problems, the most 
obvious being that you probably need more than one 
monitoring tool, since no one tool is likely to do 
everything that you need done. 


Generally, thresh has proven useful and fits 
nicely into an effective monitoring toolkit. 


Availability 


Thresh is “freely” available through http:// 
thresh.sourceforge.net/ or http://www.generalconcepts.com/ 
resources/ . 
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THRESH (1) USER COMMANDS THRESH (1 ) 


NAME 


thresh — a data-directed SNMP threshold poller 


SYNOPSIS 


thresh [ varname=value ... | 


DESCRIPTION 


thresh is a data-directed SNMP threshold poller, and uses the file system for configuration. status, and log- 
ging. Each host or device to be monitored is configured in a separate directory, using files listing SNMP 
variables, values, and a comparison indicator. 


In normal operation, thresh starts scanning a data hierarchy (as described in theshdata(5)) at a particular 
directory (set by the fopdir variable), reading DEFAULTS files, variable files, querying hosts and devices, 
and recording and reporting the results. 


thresh variables, as described in threshvars(5), and set in DEFAULTS files or on the command line, 
change thresh’s default behaviour and notification mechanisms. Any varname=value settings on the com- 
mand line override both the built-in defaults and the settings in any DEFAULTS files encountered during 
processing. 


thresh would typically be called periodically by cron(8). 


EXAMPLES 


In normal use: 
% thresh 
To use a non-default start directory: 
% thresh topdir=/some/other/place 
To traverse the data hierarchy and provide information on what would normally be queried: 
% thresh walkonly=true 
To do almost nothing: 
% thresh ’ignore=*’ 


NOTE 
thresh is written in Tcl(n), using scotty(1) and the Tnm(n) network management extensions. 

FILES 
thresh uses just about any file and directory names. The name .thresh is reserved for naming the subdirec- 
tories used by thresh to store status and logging information. 
Any files matching .threshJog_* are log files, which will grow without bound, and which you should 
arrange to rotate, archive, or truncate periodically. 

BUGS 
thresh currently only works with SNMP VI. The logging to syslog(3) should be internalized in some way, 
and not depend on logger(1). There should be some mechanism for “including” one file from another, to 
reduce the dependance on symbolic links for sharing files. thresh 1s unlikely to scale to handle arbitrarily 
large networks. 

SEE ALSO 
threshdata(5) 
threshvars(5) 
scotty(1) 
Tel(n) 
Tnm(n) 

AUTHOR 


John Sellens 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


125 


Thresh — a Data-Directed SNMP Threshold Poller Sellens 


THRESHDATA (5) FILE FORMATS THRESHDATA (5) 
NAME 

threshdata — thresh data hierarchy description 
DESCRIPTION 


The thresh(1) SNMP poller uses a configuration hierarchy to direct its actions, maintain its status informa- 
tion, and store its logs. 


Each directory in the configuration hierarchy (under the topdir directory) 1s assumed to relate to a network 
host or device, or to an intermediate name in a DNS naming hierarchy. 


NAMING 

By default, the topdir directory is assumed to refer to a device named ‘”’ — the empty string. Each direc- 
tory below topdir normally adds one more element on the right hand end of a DNS name. For example, 
below topdir, the directory 

org/usenix/conference 
is related to the DNS sub-domain ‘‘conference.usenix.org’’, and the directory 

org/usenix/conference/ts 1 
is related to the device “‘ts].conference.usenix.org”. This naming relation can be overridden by the use of 
the name variable. 


FILES 
Each directory may contain a DEFAULTS file, which contains variable settings (see threshvars(5)) that 
apply to that directory, and to all directories below that point, unless overridden on the command line or by 
other, lower, DEFAULTS files. 


Any other files found in a directory (other than those ignored by the baseignore and ignore variables) are 
assumed to contain a list of SNMP variables to monitor. thresh uses sub-directories named .thresh to store 
status and log information. 


699 


thresh data files consist of zero or more lines in the following format: 
<ws>type<ws>varlable-or-OID<ws>value<ws> 
<ws># comment ... 
<ws> 

where “‘<ws>”’ indicates white space. 


The data file elements are defined as follows: 
type A single capital letter indicating the comparison to be made in determining “normal”’. 


C Changeable — the variable’s value may change, but should be reported each time it 
changes. This is useful for semi-static data, or for monitoring things such as device inter- 
face status changes. 


G Greater than — the variable is reported if its current value is less than or equal to the spec- 
ified value. 
I Increasing ~— the variable is reported if its current value is less than its previous value. 


This is handy for watching for reset times, such as the “system.sysUpTime.0”’ variable 
resetting when a device reboots. 


L Less than — the variable is reported if its current value is greater than or equal to the spec- 
ified value. 
S Static — the variable is reported if its current value is not equal to the specified value. If 


no value is specified, then it is compared against the first-retrieved value of the variable. 
This is useful for monitoring things that should never change, such as “system.sys- 
Name.0”’. 


V Variable — the value can be anything, but it is queried and tracked to allow for later inves- 
tigation or review. 
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THRESHDATA (5 ) FILE FORMATS 


variable-or-OID 
An SNMP variable name (or name fragment that scotty(1) can interpret) or SNMP OID (e.g. 


1.3.6.1.2.1.1.3.0) to be monitored. 
value A value for the comparison. Required for G and L, optional for S, and not allowed for C, I, and V. 


SEE ALSO 
thresh(1) 
threshvars(5) 


AUTHOR 
John Sellens 
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THRESHMAIL (1) USER COMMANDS THRESHMAIL (1) 
NAME 
threshmail — mail notifier for thresh messages 
SYNOPSIS 
threshmail recipient ... 
DESCRIPTION 


threshmail expects notification messages from thresh(1) on its standard input, which it appropriately 
reformats into a mail message, and sends to every recipient given on the command line. 


threshmail would typically be set as the notifier in a thresh DEFAULTS file. 


AUTHOR 
John Sellens 
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THRESHVARS (5 ) FILE FORMATS THRESHVARS (5) 
NAME 

threshvars — configuration variables understood by thresh 
DESCRIPTION 


thresh(1) understands and observes certain configuration variables. Those variables can be provided on the 
command line or in files named DEFAULTS within the thresh data hierarchy. 


Variable names are case sensitive and are expected to be in lower case letters. 


DEFAULTS FILES 
DEFAULTS files consist of zero or more lines 1n the following format: 
<ws>VAR NAME<ws>=<ws>VALUE<ws> 
<ws># comment ... 
<ws> 
where “‘<ws>”’ indicates white space. Values can contain embedded blanks. 


Variables set by a DEFAULTS file apply at that level of the data hierarchy and below, unless overridden on 
the command line or in a DEFAULTS file further down the tree. 


GENERAL VARIABLES 
debug Generate debugging output. 
Boolean. Default: true 


verbose Generate informational messages. 
Boolean. Default: true 


walkonly 
Walk the data tree, describing the hierarchy, but not querying, reporting, or logging. 
Boolean. Default: false 


topdir The top of the data hierarchy. 
Default: /usr/local/thresh 


name The DNS name or partial name of the device or hierarchy represented by the current directory in 
the data hierarchy. Gets extended by the name of each directory as thresh descends down the 
hierarchy, but can be overridden in a DEFAULTS file. 
Default: 


baseignore 
The base list of “glob” patterns of file and directory names to ignore in the data hierarchy. If you 
override this variabJe, make sure that you end up ignoring the .thresh status directories. 
Default: . ...* CVS RCS README README.* DEFAULTS core *.core 


ignore The extended list of “glob” patterns of file and directory names to ignore in the data hierarchy. 
Having two variables makes it easy to augment the default list of names to ignore without overrid- 
ing the base list. Note that setting 
ignore = * 
will cause the hierarchy rooted at that location to be excluded from all processing. 
Default: 


prune If true, do not process further down this hierarchy if the current node is unreachable. This is 
essentially the equivalent of setting 
ignore = * 
if the current node is unreachable. This is useful, for example, for limiting the error messages that 
are generated if a gateway router is unreachable. 
Boolean. Default: false 
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THRESHVARS (5) FILE FORMATS THRESHVARS (5 ) 


SNMP VARIABLES 
community 
The SNMP V1 read community string to use to query hosts and devices. 
Default: public 


mib Specifies a file name that contains an SNMP MIB that will immediately be read and compiled into 
the running program. 


Default: 

timeout How long to wait for a response to an SNMP get request, in seconds. 
Default: 10 

retries Number of times to retransmit an SNMP get request during the timeout interval. 
Default: 5 


NOTIFICATION VARIABLES 
notifier Pipe notification messages to this program, often a mailer or mail interface. 
Default: /bin/cat 


frequency 
Minimum time before sending another identical notification message, in minutes. 
Default: 30 


describe 
Whether or not to include a variable’s MIB description field in notification messages. 
Boolean. Default: true 


msgformat 
A printf-style format string used to print notification messages, with the (cryptically named) vari- 
ables smnpvar, message, complabel, compval, newlabel, newval, desc. This could use a little more 


sophistication. 
Default: %s: %s\n %s %s\n %s %s%s\n 
LOGGING VARIABLES 
log Write out of spec entries to a log file. 


Boolean. Default: true 


logger A command like logger(1) that writes to syslog(3). 
Default: /usr/bin/logger 


syslog A syslog facility.ievel pair, as accepted by the logger(1) command, such as “‘locall.info”’. If set, 
out of spec entries are piped to the /ogger command. 
Default: 


syslogtag 
Tag to use on syslog’d entries. 
Default: thresh 


SEE ALSO 
thresh(1) 
threshdata(5) 


AUTHOR 
John Sellens 
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eEMU: A Practical Tool and 
Language for System Monitoring 
and Event Management 


Jarra Voleynik — eEMUconcept Pty Ltd 


ABSTRACT 


This paper describes a new monitoring and event management concept. eEMU 1s a client- 
server system that provides for rapid development of monitoring agents. This is thanks to its 
messaging language that takes advantage of heunstic algorithms implemented 1n the eEMU server. 
As opposed to SNMP and other static monitoring methodologies used in system and application 
monitoring, eEMU takes a whole new approach by incorporating the dynamic aspect of 
application and system events into the very heart of the dynamic message processing server. 


Motivation 


The society and industries are increasingly 
becoming dependent on computers. Consequently, 
system and application uptime management 1s becom- 
ing a critical issue. Performance of outsourcing com- 
panies is measured by SLAs (Service Level Agree- 
ments). With the advent of e-commerce on a global! 
scale, ISPs are becoming 24x7 environments as well. 
It all calls for good, flexible and reliable monitoring 
tools that would allow to proactively keep uptimes at 
the highest level and at the same time measure perfor- 
mance of [T departments. 


Monitoring tools must be simple, cross-platforn, 
scalable and cost-effective. This, to my observation, is 
not the case with most available offerings. Utilities 
such as Big Brother are widespread since they are 
cost-effective open-source alternatives and they are 
relatively simple to use. Nevertheless, alarm presenta- 
tion and agent technology in Big Brother does not 
bring any new major features compared to other 
expensive commercial offerings. 


Enterprise offerings such as Tivoli, OpenView, 
BMC Patrol and Unicenter TNG use complex multi- 
level drill-down alarm representations. Multiple win- 
dows must be opened in order to identify the cause of 
an alarm, At the top level, alarms are usually repre- 
sented with color-coded icons. Big Brother has bor- 
rowed this concept and offers color-coded button 
matrix for monitored resources with a drill-down 
option to find out more about some alarms. eEMU 
replaces color-coded icons with a simple intuitive 
interface called eEMU browser. Unlike other event 
viewers, the eEMU browser displays a single textual 
message for each event, not every event occurrence. 
Subsequent updates to the event merely update the 
existing message. As a result, a complete view of the 
enterprise is facilitated with a single message screen 
without a need to drill down for more information or 
consult the event log. By default, the eEMU browser 
displays only resources in “‘alarm state’’. It is based on 
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the premise that “‘normal state’’ information is 1n most 
cases of little practical benefit. 


eEMU was designed with modularity and inte- 
gration in mind. Through scripting hooks to the event 
engine, integration with other management tools, such 
as OpenView or Unicenter TNG, 1s simple to achieve. 


eEMU was developed by Jarra and Anna 
Voleynik of eEMUconcept after many years of experi- 
ence on large datacenter sites. Its design was an effort 
to overcome shortcomings of other monitoring offer- 
ings and provide a simple, powerful tool. Around 80% 
of monitoring needs are application or site specific. 
Standard monitoring agents that come with other ven- 
dor’s offerings will not monitor applications using off- 
the-shelf agent configurations. A well monitored site 
incorporates plenty of customisations that should be 
simple to make. Here, eEMU comes to the rescue by 
providing a message type arsenal that greatly simpli- 
fies the design of monitoring agents. In the resource 
ID section I will touch on the flexibility with which 
eEMU can process resource hierarchies. To date, 
eEMU has been providing the simplest agent develop- 
ment kit available, thanks to the fact that the alarm sta- 
tus 1s maintained by the eEMU server rather than by 
the agent. Most eEMU agents are so simple and small 
that they have been coined micro-agents. 


How eEMU works 


The majority of the currently available tools are 
based on the premise that the agent “remembers” the 
state of resources it monitors. Consequently, agents 
are relatively complex programs that run as daemons 
and maintain either a status database or status files. 
eEMU works on the premise that all the status infor- 
mation 1s handled by the eEMU server. eEMU agents 
are simple scripts or programs that use the emsg pro- 
gram to send messages to the server. The emsg pro- 
gram is a simple executable that uses the eEMU proto- 
col. The protocol is designed to deliver the necessary 
information to the server so that the server can 
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maintain the status of monitored resources. The key 
attributes of each message are resource ID, time-to- 
live, and message type. There are multiple message 
types that implement the eEMU messaging language. 


The eEMU server stores current alarms in an in- 
memory database. Alarm messages sent by agents are 
matched against the database contents to find out if the 
status of an alarm has changed. If so, the alarm status 
is updated. Each status change triggers an action 
script. For the action script, message attributes are 
facilitated as environment variables that can be used to 
take conditional actions or perform correlation of mes- 
sages. 


SNMP monitoring systems are known to produce 
UDP storms. This is partly caused by the fact that 
SNMP traps are not guaranteed to be delivered, there- 
fore the SNMP manager must regularly poll for 
resource status as well. eEMU has been tested to eas- 
ily monitor 100 systems on a 33 Kbps dialup line. This 
is due to the fact that the OK status is not transimitted 
and there is no polling necessary by the eEMU server. 
eEMU uses the TCP reliable protocol and any mes- 
sage delivery problems are quickly revealed through 
missing heart beats from the heart beat agent, which is 
an important part of every eEMU agent kit. EEMU has 
been tested to process 1000 messages a minute on a 
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400 MHz Pentium PC. On a large site with hundreds 
of nodes it may be easier to deploy 2 or 3 tiers of mon- 
ituring servers with a consolidation server at the top. 
Messages from middle tier servers can be selectively 
forwarded to the higher level server. To make a rea- 
sonable effort of delivering messages, emsg uses an 
exponential back-off for spacing out several connec- 
tion attempts before giving up and retuming a failure 
code. Consequently, out-of-band communication 
channel can be used by agents if the primary commu- 
nication channel is down. 


The user interface is represented by an intuitive 
event console called eEMU browser. The browser uni- 
fies the best of both worlds - drill-down icons and 
scrolling event screens. To enhance user communica- 
tion, the browser allows the operators to not only 
annotate existing messages but send new ones as well. 
Technically, there is no difference between messages 
sent by scripting agents and messages send by opera- 
tional staff. The whole purpose is to create an event 
flow between enterprise components and staff. 


Figure 1 shows a list of alarms that are color- 
coded by severity for easy interpretation. The “user 
root is logged in” message is annotated with a change 
request number to indicate a scheduled work on the 
firewall. 
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Figure 1: Color coded alarms. 
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Figure 2 shows the business view mode with 
alarms logically grouped for a consolidated functional 
view of the enterprise. Alarms under a specific busi- 
ness group can be displayed by clicking the business 
group button. 


Resource ID and Message Class 


In order to identify monitored resources, each 
resource is assigned an Object ID. For a simple Object 
ID processing, the Object ID must form the first word 
of each message. To uniquely identify a resource, we 
need a node name as well. Consequently, “node_ 
name”’:“Object ID” form a Resource ID, which 1s a 
message key maintained by the eEMU server for 
every alarm. All references to existing alarms are done 
based on the Resource 1D. Examples of Resource IDs 
are: web.dumbo.com.au:/usr or ps.porky.com.au ‘printer. 
sydney.accounts 1. 


If a new agent is written, its messages will be 
sent with its own resource [D. Therefore, the resource 
ID should fit into the existing resource ID hierarchy, 
just like SNMP OIDs are constructed based on a hier- 
archical principle. With eEMU there is no need to reg- 
ister the resource ID since the server doesn’t need to 
know it in advance. !t makes deployment of new 
agents and resource hierarchies an extremely simple 
task compared to SNMP agents. Today, systems and 
applications undergo a constant change, thus repre- 
senting a very dynamic environment that calls for an 
equally dynamic monitoring tools. 


If we start integrating eEMU with a call logging 
system, we will find that calls need to be logged once 
only for a specified resource problem. Later messages 
for that particular resource need to update the existing 
call only. Resource ID makes call logging possible. 


Resource IDs are part of eEMU agents and can 
be part of the agent code or part of the agent configu- 
ration. 


Time-to-Live 


Time-to-live is a critical attribute of each mes- 
sage. It determines how long the message is kept in 
the status database maintained by the server. Let us 
demonstrate the time-to-live on a simple agent sce- 
nario. A filesystem agent runs every 5 minutes. It 
scans all the mounted filesystems and compares their 
utilisation with predefined thresholds. If a threshold is 
exceeded, a message to that effect is sent to the eEEMU 
server. Time-to-live attached to the message will be set 
to 7 minutes. The server will store the message in the 
status database for the period of 7 minutes. If the mes- 
sage is not updated within 7 minutes, the server will 
delete it. The next filesystem agent run will occur 5 
minutes later. It will send another message to the 
server if the offending filesystem’s threshold is stil] 
exceeded. Since the previous message 1s still stored in 
the database (with 2 more minutes to live), the refresh 
message will re-set its life to 7 minutes. The situation 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


eEMU: A Practical Tool and Language for System Monitoring and Event Management 


repeats itself as long as the filesystem’s threshold is 
exceeded. Once the problem is fixed, the message 
times-out and is removed from the status database by 
the server. 


Time-to-live concept allows to transfer alarm sta- 
tus management responsibilities onto the server, thus 
tremendously offloading eEMU monitoring agents. 
The concept of time-to-live can also be used in the 
reverse sense as implemented in the sleep message 
type, described below. 


Message Class 


The message class attribute was designed for 
flexible classification of resources, building business 
rules, event logging and escalation. The class can be 
used to hierarchically build classes or trees of 
resources and group them. The class attribute can 
decide which support group needs to know about the 
event and subsequently take action. Also, it may 
describe a resource hierarchy from the application or 
system point of view. Thus, class assists in both, the 
business escalation process (it adds a business layer to 
the message) and resource identification. Since the 
class attribute can be retrieved from within action 
Scripts, it is trivial for administrators to implement 
conditional message processing based on the class 
string. 

Message classes are part of eEMU agents and 
can be part of the agent code or part of the agent con- 
figuration. 


Class examples are: /UNIX/PRD/FS, /NT/DEV/ 
ORA. 


eEMU Messaging Language 


Analysis of monitoring agents behavior revealed 
that there are typical scenarios that repeat themselves. 
The findings prompted us to tum these scenarios into 
event message types, which resulted in the eEMU 
messaging language. 

The eEMU messaging language and time-to-live 
are the key ingredients for making eEMU so simple to 
use. A fairly complex monitoring scenario can be 
expressed in a few lines of code. 


Currently eEMU supports ten types of messages: 
normal, delete, count, sleep, wakeup, event, mask, 
query, comment and control. 


The normal message is held in the status 
database for the time-to-live penod, whereupon it 1s 
removed from the database if it is not refreshed. By 
setting time-to-live to slightly more than the agent 
polling interval, the message is guaranteed to stay in 
the status database for the duration of the alarm, thus 
reflecting alarin status. 


The delete message can be sent to the server to 
delete a message. 


The count message behaves similar to the nor- 
mal message only it is in an inactive state until the 
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message has been refreshed a predefined number of 
times. Each count message is accompanied with a 
“lag” value that indicates how many successive alarm 
messages necd to be received before the resource sta- 
tus is changed. As an example, a system administrator 
is not usually interested us CPU reaching 100% for 
short periods of time. It becomes a problem only if the 
CPU utilisation 1s high consistently over longer peri- 
ods of time. There are many resources exhibiting simi- 
lar qualities, e.g. the swap space, network traffic and 
memory utilisation. Count messages are a powerful 
means for monitoring such resources. 


The sleep message is kept inactive (sleeping) in 
the status database. It becomes active when it times 
out. Of course, it can be removed from the database 
with a delete message before the actual time-out. The 
premature sleep message removal ts used in many sce- 
narios such as backups. When a backup is started, a 
sleep message is sent to the eEMU server. Time-to- 
live of the sleep message ts slightly greater than the 
maximum duration of the backup. At the end of the 
backup, the sleep message is deleted. There are two 
conditions under which the backup raises an alarm. 
The first is that the backup script failed or crashed. 
The second is that the backup is running overtime. 
Both conditions will be revealed by the sleep message 
timing out and becoming active. 


One of the most interesting uses of the sleep 
message is a heart beat agent that monitors system 
uptime. E.g. the heart beat agent on node “dumbo” 
keeps sending heart beat messages, such as 

S$ emsg -o sleep -n eemuserver \ 
-p 1965 -t 7m -s 1 -c /UNIX/HB \ 
-w icecream -m "heart-beat is down" 


The above message Is sent to “eemuserver”’ lis- 
tening on port 1965, time-to-live of 7 minutes, sever- 
ity 1, class /UNIX/HB, password “icecream” and 
message “heart-beat is down’’. The resource ID is 
‘‘dumbo:heart-beat”’. 


With heart beat agents, no node discovery is nec- 
essary. The first heart beat registers the node with the 
eEMU server. If the node is down, the heart beat mes- 
sage is not refreshed. The server turns the sleep mes- 
sage into an active message that triggers an action and 
displays the alarm on the eEMU browser. Once the 
problem is fixed, the message goes to sleep again by 
switching its status to inactive. After giving some 
thought to the heart beat concept one finds out that it 
is completely self-maintained. Frequent adding of new 
nodes to a large monitored base is as easy as sending 
the first heart beat without touching the eEMU server 
configuration. Also, removing a node is as simple as 
deleting the heart beat sleep message from he eEMU 
database. Experience from large scale eEMU deploy- 
ments has shown that heart beat agents are an 
extremely reliable means of detecting problems. 


The wakeup message 1s used to wake up a sleep 
message. Let us suppose that there 1s a sleep message 
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already in the database but we would like to expedite 
its time-out. Jt can be accomplished with a wakeup 
message. 


The event message behaves just like the normal 
message with the difference that by default it does not 
show on the eEMU message browser. It is predomi- 
nantly used to notify the eEMU server of an event and 
subsequently keep the event in the database for the 
event’s duration. Event messages trigger action 
scripts and many times they are used Just for that. At 
other times, a query message (described below) 1s 
used to query the status database for a particular event. 
E.g., a batch job on system A sends an event message 
to eEMU on successful completion. System B runs its 
batch job only on condition that the batch job on sys- 
tem A completed successfully. Before the batch job on 
system B runs, it first checks with a query message if 
an event that designates the system A batch job’s suc- 
cessful completion exists. 


The mask message instructs eEEMU to mask out 
messages of a particular resource ID. The effect of the 
mask message 1s set for a time-to-live interval. It lends 
itself to scenarios such as when a backup shuts down 
an application that is being monitored. The application 
alarms can be masked out for the duration of the 
backup. 


The query message has two main uses. The for- 
mer use lies in querying the status database to find out 
about other alarms, This is handy for cross-system 
event correlation. The latter use lies in downloading 
files from the eEMU server. Central storage of files on 
the server can be used for software distribution pur- 
poses among others. 


The comment message is used to annotate exist- 
ing eEMU messages. IT staff can instantly provide 
information related to raised alarms, such as actions 
taken to correct existing alarms or reasons for sched- 
uled downtimes. 


The control message is used for administrative 
purposes such as suspending the eEMU server. 


eEMU Agents 


eEMU agents have the eEMU messaging lan- 
guage at their disposal. The core of the agent ts the 
emsg executable. In many cases, the agent code Is a 
few lines of code. This is due to the intelligence built 
into various message types eEMU makes available. 


Most monitoring systems rely on thick agents 
that maintain status of monitored resources. The agent 
technology can be tapped into by using a provided 
SDK (Software Development Kit). Apart from a need 
to have programming skills, administrators are bound 
to learn the API and develop agents that are not 
portable, depend on one vendor and are hard to main- 
tain. With eEMU, the agents are typically simple 
scripts in a language of the administrator’s choice. 
Due to the agent’s simplicity and scripting language 
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implententation, they are portable, developed in a 
rapid time-frame and easy to maintain. Moreover, 
eEMU agents can be simply scheduled through the 
UNIX cron or other scheduler, thus further simplifying 
the agent code. 


To demonstrate eEMU features, below are a few 
scenarios and the way they can be tackled with eEMU. 


A batch job run takes more than 10 minutes. If it 
is less than 10 minutes, the job terminated prematurely 
and we want to know about it. The following is the 
eEMU wrapper code: 

emsg -n eemuserver -o "count 2" \ 
=) LOm:c¢" 4 VMS/ BATCH. \ 
=m. “batch job tailed” 
Start batch_job 
emsg -n eemuserver -o "count 2" \ 
= =: Co / VMS/ BATCH \ 
ai Daten: 2).0b.-fadtad! 


Before the batch job is started, a count message with 
the lag value of 2 is sent to the server. This message 
will be sitting in the server in an inactive state for 10 
minutes and unless it is refreshed, it will be deleted. 
Notice that the second emsg is sent for the same object 
ID, namely batch_job, whereby the lag of the count 
message is also set to 2. It means that if the first mes- 
sage 1s still in the database at the time the second mes- 
sage arrives, the message will assume an active state 
and displays in the eEMU browser. 


A cold backup of an oracle database on system 
‘““‘porky” shuts the database down. Since the database 
processes are monitored with a process agent, we are 
inevitably going to receive an alarm that the database 
is down. In case of planned downtimes, we do. not 
wish to receive alarm messages. Some monitoring sys- 
tems offer a calendar feature that can be used to tem- 
porarily stop alarms from occurring. The problem is 
that a “static” calendar configuration needs to be 
attached to the alarm. With eEMU, message masking 
is dynamically adjusted to the oracle backup interval. 

emsg -n eemuserver -h porky -o mask \ 
=t-2h-=-¢ / PROD/ORA, 4 
-m "Sap_oracle is down" 
shutdown_oracle.sh 
backup_oracle.sh 
startup_oracle.sh 
emsg -n eemuserver -o delete \ 
-m "porky:sap_oracle" 


The first emsg sends a mask message for the 
porky:sap oracle resource ID. It will mask out 
sap oracle alarm messages for 2 hours. Next, the 
database 1s shut down and a backup performed. After 
the backup has been compieted, oracle is started up 
and the mask message deleted. A bonus 1s that if the 
backup runs overtime (more than 2 hours in this case), 
we will be alerted of the application being down 
beyond the backup window. 


A CPU agent can be as small as the following 
single line: 
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eval “ymestat oe vai eae \ 
awl tae 05 16. MIG ep ra me \ 
"emsg -n eemuserv -p 1 965 \ 
=¢ LINUX/CPU <t 7m’ -s: 1) \ 
O71 Vi COUmE: 2o\i oN 
=m: \"CPU “ds 100% utilised \""))°- 


The above line retrieves the CPU idle time from 
a vmstat output and if the idle time is less than 1% it 
sends a count eEMU message with a lag of 5. If five 
such messages are received in a row, an alarm 1s 
raised. 


In order to expand the usability of eEMU, the 
query message was equipped with a file download 
option. As aresult, a file can be downloaded from the 
eEMU server. This feature can be used for software or 
script distribution but also for ad-hoc remote job exe- 
cution. The following simple task distribution agent 
regularly (from cron) checks if there is a specific 
script in the download directory on the server. If there 
is a script for download, it is downloaded and exe- 
cuted. 

RET=‘emsg -n eemuserver \ 
-p 1965 -w icecream -o query \ 
=m “PILE dumboxejob!) il -\ 
tee. j0bto_do.sh-* 
it i US RE none. chon 
chmod 700 job_to_do.sh 
job. todo-.:sh 
ea! 
In the above, the downloaded script is displayed by 
emsg on the standard output and subsequently saved 
in 3ob_to_do.sh for later execution. If there 1s no script 
for download on the server, emsg retums string 
‘‘none”’ and the body of the “if” statement is not exe- 
cuted. Notice that we do not need to log into the 
remote system to perform the task. For a site with 
many systems, the script or file distribution feature 
can provide great benefits for day-to-day system 
administration. 


eEMU Action Scripts and Automation 


Without action scripts, alarms can be viewed on 
the eEMU browser but there cannot be any automa- 
tion. One of the major benefits of monitoring systems 
is that a corrective action or alarm escalation can take 
place automatically without human _ intervention. 
eEMU action scripts can be written in any program- 
ming or scripting language. The scripts are called in 
the background, whereby message attributes are 
passed to it as environment variables. Korn shell, Perl, 
Python, Tcl or Visual Basic are examples of excellent 
scripting languages that can be used with eEMU. 


eEMU features three action scripts: input, output 
and delete. The input action script is called immedi- 
ately on receipt of a new message. If the input script 
returns a value greater than 0, the message is dis- 
carded. The input script can be used for message 
replication or to supplement security and let in mes- 
sages from specified hosts only. 
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The output action script ts called for every mes- 
sage processed by eEMU except for delete messages. 
Delete messages invoke a separate script called 
“‘delete action script’. 


An exhaustive group of environment variables is 
provided for all the three action scripts, thus alarm 
actioning can be easily coded. There is no limit on 
what can be done in action scripts. emsg can also be 
used in action scripts to query the eEMU database, 
read a file or send a new message to eEMU. Care must 
be taken to avoid infinite message loops. 


With a combination of resource IDs, message 
classes and message severity, call logging or paging 1s 
simple to achieve through action scripts. For example, 
to send a page if an alarm of severity 1 occurs, the fol- 
lowing output action script can be used: 

if [ SELSEV -e® di -a SE SG6UaT \ 
-eq 1 -a SE_TYPE = "menmal" | ;then 
/usr/local/bin¥page \ 
SE_CLASS SE_HOST SE_MSG 
£1 


The first occurrence of a normal message 
(E_TYPE) with severity 1 (E SEV) calls the 
/usr/local/bin/page script. The script can page the 
appropriate staff based on the message class 
(E_CLASS). 


Using eEMU 


While syslog and e-mail has been in use for 
some time and as such are powerful, proven messag- 
ing tools, they do not provide enough flexibility for 
dynamic monitoring in ever changing environment. 
With the messaging arsenal eEMU provides it 1s possi- 
ble to build simple powerful agents not only for the 
system and hardware, but most importantly for appli- 
cations. It is increasingly necessary to monitor various 
applications. This task is not easy with system agents 
provided with most other monitoring solutions. The 
eEMU messaging language is expressive enough to 
capture most typical scenarios in application monitor- 
ing. Such scenarios are dynamic masking out alarms, 
measuring application response times, catching over- 
time or hung business processes, identifying crashed 
scripts, synchronising batch job etc. Since eEMU 
agents do not, in most cases, have to store the resource 
status information, all that needs to be done is trans- 
form the monitoring case into the eEMU messaging 
language and if necessary, deploy the provided stan- 
dard agents, e.g. the log agent, directory agent and 
process agent. 


The eEMU server is a powerful implementation 
that has a minimum demand on operating system 
resources. E.g. the message database on disk does not 
typically grow beyond 500 kB. Action scripts are sim- 
ple to implement. Action scripting is designed for easy 
integration with other products, such as paging soft- 
ware or third party messaging. 


Status messages are equipped with a timestamp 
and written into a log file. There is one log file for 
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each day. The contents of the log file can be used for 
generating reports or ad-hoc alarm searches. Message 
attributes in the file are delimited with a vertical bar 
for handy processing by scripts or other reporting soft- 
ware. 


The eEMU browser unifies the best capabilities 
of today’s alarming concepts, namely color-coded sys- 
tem icons and event logs into one interface. As a 
result, the browser is extremely intuitive with mini- 
mum requirements on the operational staff. The latest 
version of the eEMU browser implements “‘business 
views” that allow to segment alarms into groups by 
function, geographical region, business importance 
etc. 


There are a few passwords that enable access to 
specific message type groups. For example, helpdesk 
staff can view and annotate messages but cannot 
delete them. Each message type must be equipped 
with the correct password in order to be accepted by 
the server. 


Since eEMU agents do not typically run as dae- 
mons, but rather are scheduled through a scheduling 
mechanism, such as cron on UNIX, eEMU represent a 
very safe monitoring solution for systems deployed in 
insecure environments such as firewalls. eEMUcon- 
cept provide an eEMU agent scheduler for Windows 
NT. 


eEMU Integration 


eEMU is a modular system that allows to build 
multi-tier hierarchies of monitoring agents and 
servers. The emsg executable forms the client side of 
the system. It can easily be integrated into any third 
party system. The integration can be done either at the 
emsg level or at the server level through action scripts. 


eEMU has been successfully integrated with HP 
OpenView and Unicenter TNG. They are both SNMP 
based, so how was it done ? Simply through action 
scripts. In case of standalone SNMP agents without an 
SNMP manager, there are a few approaches. An 
SNMP daemon can listen for SNMP traps and log 
them into a log file. The eEMU log agent is pointed at 
the log file and sends alarms for traps of our interest. 
If SNMP GETs need to be employed, a simple agent 
script, possibly written in perl, uses an snmpget utility 
to retrieve a resource status and subsequently sends a 
message to eEMU. 


In the above, I described the case of alarms that 
are forwarded to eEMU for display on the eEMU 
browser and for triggering alarm actions. However, it 
is also possible to forward alarms from eEMU to a 
third party system. We can use the eEMU action 
scripts and command line interface as a vehicle for 
such forwarding. 


Summary 


In the network realm, SNMP is a successful, 
firmly established monitoring methodology. As a 
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result, there have been numerous attempts to apply 
SNMP to system monitoring. Our experience shows 
that such attempts are not successful. What adminis- 
trators need is a powerful, simple command line inter- 
face to the monitoring server. Moreover, the interface 
needs to possess intelligence so that agent scripting 
requirements are reduced to the minimum. eEMU 1s a 
system that brings message type and time-to-live con- 
cepts that make the above possible. With eEMU, 
every system and network administrator is able to 
build monitoring and messaging infrastructure that 
forms the backbone of every enterprise management. 
There is no enterprise management without a powerful 
messaging foundation equipped with scripting inter- 
faces at all levels. Some vendors like to say that enter- 
prise management Is using their products or their inte- 
gration kits for other third party programs. This is not 
so. Enterprise management is based on open standard 
interoperability and easy integration hooks in and into 
existing systems. A lot of effort has been put into 
making vendors build their products to recognised 
standards. There has been more or less success doing 
that. However, command line interface to programs 
and script invocation on various events is still the best 
and most versatile API available. 


Availability and Support 


The eEMU software is available from eEMU- 
concept Pty Ltd, Australia. A free evaluation licence is 
available on request from the company’s home web 
page (http://(www.eemuconcept.com). The eEMU kit 
includes Unix and Windows NT system agents. 
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Series for Network Monitoring 
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ABSTRACT 


The open-source software RRDtool and Cricket provide a solution to the problem of 
collecting, storing, and visualizing service network time series data for the real-time monitoring 
task. However, simultaneously monitoring all service network time series of intcrest is an 
impossible task even for the accomplished network technician. The solution is to integrate a 
mathematical model for automatic aberrant behavior detection in time series into the monitoring 
software. While there are many such models one might choose, the primary goal should be a 
model compatible with real-time monitoring. At WebTV, the solution was to integrate a model 
based on exponential smoothing and Holt-Winters forecasting into the Cricket/RRDtool 
architecture. While perhaps not optimal, this solution is flexible, efficient, and effective as a tool 


for automatic aberrant behavior detection. 


Introduction 


Real time management of a service network 
infrastructure at the IAP/ISP level is not a trivial task. 
First, there is the sheer quantity of data generated on a 
minute-by-minute basis. The WebTV production ser- 
vice infrastructure consists of tens of switches and 
routers, hundreds of host computers, and thousands of 
application daemon instances to support a subscriber 
base of over 1 million users. Second, there is great 
variety in the types of data collected. The WebTV pro- 
duction service monitors SNMP counters over net- 
work links, host statistics such as CPU load and I/O 
operations, and event logs for application daemons. 
Every variable monitored, whether byte traffic on a 
switch port, CPU load of a host machine, or requests 
handled by a cookie daemon, generates a time series. 
All of these time series reflect some part of the overall 
service network health. 


The first challenge therefore, is to collect, store, 
and provide real-time access to this vast and diverse 
data. The open-source software RRDtool [6] and 
Cricket [1, 2] meet this first challenge. Using a web 
browser, a network technician can quickly navigate to 
and view a time series graph for a target and variable 
of interest. 


The network technician is likely to be interested 
in aberrant behavior; that 1s, changes in the short-term 
behavior of a time series (on the order of minutes or 
hours) that are inconsistent with past history. Long- 
term trends (on the order of weeks or months) are not 
of interest from the service monitoring perspective 
because one expects a time series to evolve in a 
dynamic environment. Aberrant behavior may indicate 
a performance bottleneck, application component fail- 
ure, or system downtime. In some cases, aberrant 
behavior is anticipated; other times it 1s not (see sec- 
tion “Defining Aberrant Behavior’’). 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


The second challenge of network monitoring is 
to automatically identify aberrant behavior in the 
midst of thousands of service network time series. 
Once such behavior is identified, an alert can be trig- 
gered to bring the technician’s attention to the poten- 
tial problem. Existing software tools provide some of 
this functionality, but these solutions usually rely on 
simple rules or thresholds (i.e., memory utilization 
should be below 80%), These rules and thresholds are 
sufficient for many applications, but they can’t detect 
more subtle changes in behavior and they apply a 
static criteria to detect aberrant behavior rather than a 
dynamic one. 


This paper describes a partial solution to this sec- 
ond challenge of network monitoring at the IAP/ISP 
level. Section “‘Description of the Model’’ discusses 
the aberrant behavior detection algorithm, with a focus 
on understanding the algorithm parameters. The bulk 
of the software implementation is in RRDtool, which 
is the focus of section “Enhancements to RRDtool.”’ 
Section “Enhancements to Cricket” discusses details 
relevant for Cricket. The conclusion lists availability 
of the software. 


Defining Aberrant Behavior 


Suppose a statistical model exists that describes 
the behavior of a time series (or at least the character- 
istics of interest). With such a model, one can define 
aberrant behavior as behavior that does conform to the 
model (or is not well described by the model). 


Of course, aberrant behavior with respect to a 
statistical model may or may not reflect a real event of 
interest for the technician. In the case that it does not, 
it 1s a false positive. Obviously, the ideal is to mini- 
mize the rate of false positives while identifying all 
events of real interest. However, this ideal can rarely 
be achieved. In most detection systems, there is a trade 
off between selectivity (avoiding false positives; also 
referred to as specificity and precision) and sensitivity 
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(ability to detect true positives; also referred to as 
recall). While it is important to remain cognizant of 
these issues, they become less important if one per- 
ceives a Statistical model for aberrant behavior as a 
screening mechanism rather than a surrogate for the 
expert judgement of a network technician. 


Note that this definition treats each time series 
independently of all others. No doubt there is much to 
be gained by leveraging the relationships between ser- 
vice network variables, but that 1s not addressed in this 


paper. 
Description of the Model 


Model Design Goals 


Many service network variable time series 
exhibit the following regularities (characteristics) that 
should be accounted for by a model: 

1. A trend over time (i.e., a gradual increase in 
application daemon requests over a two month 
period due to increased subscriber load). 

2. A seasonal trend or cycle (1.e., every day bytes 
per second increases in the morning hours, 
peaks in the afternoon and declines late at 
night). 

3. Seasonal variability. (1.e., application requests 
fluctuate wildly minute by minute during the 
peak hours of 4-8 pm, but at 1 am application 
requests hardly vary at all). 

4. Gradual evolution of regularities (1) through 
(3) over time (i.e., the daily cycle gradual shifts 
as the number of evening daylight hours 
increases from December to June). 

This list is by no means exhaustive; but it captures the 
most important characteristics. 


In addition to modeling time series regularities, 
model design must consider the real-time monitoring 
context. Complicated statistical models are unlikely to 
be understood by network technicians and unlikely to 
be feasible computationally in a real-time context. 


Overview of the Model 


Aberrant behavior detection is decomposed into 
three pieces, each building on its predecessor: 

e An algorithm for predicting the values of a time 
series one time step into the future. 

e A measure of deviation between the predicted 
values and the observed values. 

e A mechanism to decide if and when an 
observed value or sequence of observed values 
is ‘too deviant’ from the predicted value(s). 


The proposed model is an extension of Holt- 
Winters Forecasting, which supports incremental 
model updating via exponential smoothing. The fol- 
lowing sections discuss the model in some detail and 
require some mathematical notation. Let 
Vi-'*¥-t>Ve Yee) °** denote the sequence of values 
for the time series observed at some fixed temporal 
interval (recall RRDtool maps an irregular time series 
to a regular interval). Let m denote the period of the 
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seasonal trend (1.e., the number of observations per 
day). 
Exponential Smoothing 


Exponential smoothing [3] is a simple algonthm 
for predicting the next value in a time series given the 
current value and the current prediction. Let j,,, 
denote the predicted value for time ¢+ 1, then: 
Visi = Or + Ck ayy, 

The prediction is actually a weighted average of 
all past observations in the time series. The premise of 
exponential smoothing is that the current value is most 
informative for prediction of the next value, and that 
the weight of older observation decays exponentially 
as the observation moves further into the past. It 1s an 
incremental algorithm because the next prediction is 
obtained by updating the current prediction with the 
current observed value. 


a is the model parameter and 0 <a < 1. It deter- 
mines the rate of decay (1 — @) and the weight the 
current value is given during the incremental update. 


The Holt-Winters Forecasting Algorithm 


Holt-Winters Forecasting [3] is a more sophisti- 
cated algorithm that builds upon exponential smooth- 
ing. Holt-Winters Forecasting rests on the premise that 
the observed time series can be decomposed into three 
components: a baseline, a linear trend, and a seasonal 
effect. The algorithm presumes each of these compo- 
nents evolves over time and this is accomplished by 
applying exponential smoothing to incrementally 
update the components. 


The prediction is the sum of the three compo- 
nents: 
Very = Ay + by + Crp iam 
The update formulas for the three components, or 
coefficients a, 5, c are: 
e Baseline (“intercept”): 
a, =A.) + lS O)G@er tle) 
e Linear Trend (“‘slope”’): 
b, = Bla, — a,~1) + (1 — B)by-1 
e Seasonal Trend: 
= V0, 8) te ye, 
As in exponential smoothing, the updated coefficient 
is an average of the prediction and an estimate 
obtained solely from the observed value y,, with frac- 
tions determined by a model parameter (a, f, y). 
Recall m is the period of the seasonal cycle; so the 
seasonal coefficient at time ¢ references the last com- 
puted coefficient for the same time point in the sea- 
sona| cycle. 


The new estimate of the baseline is the observed 
value adjusted by the best available estimate of the 
seasonal coefficient (c,_,,). As the updated baseline 
needs to account for change due to the linear trend, the 
predicted slope is added to the baseline coefficient. 
The new estimate of the slope is simply the difference 
between the new and old baseline (as the time interval 
between observations is fixed, it is not relevant). The 
new estimate of the seasonal component is the 
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difference between the observed value and the corre- 
sponding baseline. 


a, B, and y are the adaptation parameters of the 
algorithm and 0<a, $, y< 1. Larger values mean 
the algorithm adapts faster and predictions reflect 
recent observations in the time series; smaller values 
means the algorithm adapts slower, placing more 
weight on the past history of the time series. 


Note that the update formulas imply an imple- 
mentation need only to store the current values of the 
slope and intercept, and a single period of seasonal 
coefficients, as these stored values are replaced at each 
iteration. 


Holt-Winters Forecasting can also predict a time 
series further than a single time step in the future [3]. 
This multi-step prediction provides a mechanism to 
handle missing data. 


Confidence Bands: Measuring Deviation 


Confidence bands measure deviation for each 
time point in the seasonal cycle; this mechanism mod- 
els seasonal variability. The measure of deviation is a 
weighted average absolute deviation, updated via 
exponential smoothing: 

ad. =ybr1 —Vl +O — dm 
Here d, is the predicted deviation at time step ¢. The 
update formula for d, is similar to that of c,. They even 
share the same adaption parameter, y. The confidence 
band is. simply the _ collection of intervals 
(Vi 0 2 td. J, + On G4.) Aor each time pomt 
y, in the series. 

6, and 6_ are scaling factors for the width of the 
confidence band. Often, a symmetric confidence band 
is desired and 6, =0_. In this case, denote the com- 
mon parameter 0. Given some assumptions and statis- 
tical distribution theory, sensible values of 6 are 
between 2 and 3 [7]. 


Aberrant Behavior Detection 


A simple mechanism to detect an anomaly is to 
check if an observed value of the time series falls out- 
side the confidence band. However, this mechanism 
often yields a high number of false positives. A more 
robust mechanism is to use a moving window of a 
fixed number of observations [7]. If the number of 
violations (observations that fall outside the confi- 
dence band) exceeds a specified threshold, then trigger 
an alert for aberrant behavior. 


Formally, define a violation as an observation y, that 
falls outside the interval: 
(Y, =O : Dm y, + 0 ; dizm) 

Define a failure as exceeding a specified number of 
threshold violations within a window of a specified 
numbers of observations (the window length). 
Temporal Smoothing of Seasonal Cycle and Varia- 

tion 

As discussed thus far, each component of the 


seasonal coefficients vector is determined indepen- 
dently. It seems reasonable to assume that the seasonal 
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effect is a smooth function over the period, not a dis- 
continuous series of points. A similar argument 
applies to the seasonal deviations. Note that exponen- 
tial smoothing performs smoothing across seasonal 
cycles, but does not perform temporal smoothing 
within a seasonal cycle. 


At a cost of adding some additional overhead to 
the implementation, the model performs temporal 
smoothing within a cycle for the seasonal coefficients 
and deviations. The smoother used is an equal-weight 
moving average, with a window of 0. 05m. 


Choosing Model Parameters 


The model parameters need to be set and tuned 
for the model to work well. There is no single optimal 
set of values, even restricted to data for a single vari- 
able. This is due to the interplay between multiple 
parameters in the model. 


For example, consider two observations in 
sequence, y, and y,,,. The intercept (a), slope (6), and 
seasonal (c) coefficients all ‘absorb’ some part of the 
difference between y, and y,,, during the exponential 
smoothing update. It 1s safe to assume some of the dif- 
ference 1s noise, so updates to the coefficients need not 
account for all of the difference between y, and y,,). 
The values of a, 8, and y determine the relative share 
of the difference assigned to a changing baseline, a 
changing linear trend, and a changing seasonal coeffi- 
cient 


Here are some common sense guidelines for set- 
ting parameters: 
ea: At least one of a, B, and y should allow 
adaptation in a short time frame. As seasonal 
updates occur infrequently for each coefficient 
(once per cycle), and the goal of £ is to capture 
a slowly changing linear trend, the most logical 
choice 1s a. Use exponential smoothing weights 
to make an educated choice for a. The sum of 
the most recent n weights is 1 —(1 — a)” and 
of course the sum of all weights is 1 (ignoring 
Initialization, see section ‘‘Initialization’’). 
These facts can be manipulated to choose @ 
using the formula: 
fog( | — total weights as % ) 
a=1—-e)|—_—_———————— 
# of time points 
fog() denotes the natural logarithm. For exam- 
ple, if one wants observations in the last 45 
minutes to account for 95% of the weights, and 
observations occur at five minute intervals 
(nine time points), then the formula yields 
a = 0.28. This formula can be rearranged using 
simple algebra to compute either the total 
weights as a percentage or the number of time 
points (elapsed time). For example, if a = 0.1, 
then the most recent hour of observations at 
five minute intervals (12 time points) accounts 
for 75% of the baseline prediction. 
B: As the purpose of # is to capture a linear 
trend longer than one seasonal cycle, it is 
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logical to choose f such that one seasonal cycle 
does not account for a majority of the exponen- 
tial smoothing weights. The formula discussed 
previously applies with # replacing a. For 
example, if the period of the cycle is one day at 
one observation every five minutes (288 per 
day), then setting 6 =0.0024 will guarantee 
that observations within the last day account for 
less than 50% of the smoothing weights. 
e y: The seasonal adaptation parameter can also 
be selected using exponential smoothing 
weights using a variation of the previous for- 
mula. Note this single parameter controls both 
seasonal coefficient and deviation adaptation, 
on the assumption that seasonal trend and vari- 
ability evolve together over time at roughly the 
same rate. 
0: As noted in confidence bands section, the 
scaling factor of the confidence bands can be 
chosen by appealing to statistical distribution 
theory. Reasonable values fall in the interval [2, 
3]. Choose 2 to detect more failures (which 
may just mean a higher rate of false positives). 
Window length and threshold: Given the goal 
of real-time monitoring, the window length 
should be at most on the order of an hour (1.e., 
for five minute intervals, choose a window 
length between 9 and 12). A higher threshold 
will make the model robust to false positives, 
but perhaps at the cost of missing true failures. 
These parameters are probably the most diffi- 
cult to set a priori. 


Initialization 


The model requires initial values for the inter- 
cept, slope, seasonal coefficients, and deviations (sea- 
sonal variability). These could be set arbitrarily, com- 
puted from a long history of past data, or bootstrapped 
from data as it becomes available. The implementation 
in RRDtool is to bootstrap the algorithm from a cold 
Start. 


Initial values exert influence for some time. The 
analysis of exponential smoothing weights in the pre- 
vious section assumes that influence of initial values 
has become negligible. For the intercept coefficient, 
the weight of the initial value in exponential smooth- 
ing for observation ¢ is (| —a)'~'. Similar formulas 
hold for 8 and y. These formulas can be used to calcu- 
late the influence of initial values. For example, if the 
seasonal period is one day, 10 days have elapsed since 
initialization, and y = 0.1, then the weight of the ini- 
tial value in the predicted seasonal coefficient is 0.39. 
In contrast, the weight of the most recent observation 
(which in the long run is the most influential) is only 
0.1. 


Alternatives 


While the model is designed to meet several 
goals, it is not optimal. The proposed algorithm lacks 
a formalism found in some other models. It is cer- 
tainly true that there is no uniformly superior 
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forecasting algorithm for all time series, but consider 

the comments of researcher Richard Lawton [5]: 
The Holt-Winters method is one of the best 
known forecasting techniques which allows the 
seasonal pattern to adapt over time... When 
compared with other methods the technique 
has been found to perform relatively well and it 
has the merit of being understood by users who 
lack a statistical background without sacrific- 
ing the ability to adapt to changing patterns in 
the data. 


Enhancements to RRDtool 


RRD is the acronym for Round Robin Database. 
RRD is a system to store and display time-series data 
[6]. It stores the data compactly, minimizes I/O opera- 
tions for real-time updates, and presents useful graphs 
by processing the data at different temporal resolu- 
tions. 


This section describes the implementation and 
usage of aberrant behavior detection in RRDtool. 
Some familiarity with the internals of the current 
release (1.0.x) of RRDtool is assumed, as this section 
makes reference to the pre-existing architecture. 


Motivation 


There are several reasons why support for aber- 
rant behavior detection is integrated within RRDtool, 
as opposed to implemented as a standalone program. 
These include: 

e Facilitates efficient real-time application of 
aberrant behavior detection. An external pro- 
gram would have fetch data from the RRD at 
the same frequency of update, while code 
within RRDtool is guaranteed to operate on this 
data already in-memory. Efficiency is a top pri- 
ority for the service network at the IAP/ISP 
level, where RRDtool can be essential part of 
the monitoring system of hundreds of network 
interfaces and application services. 

Leverages ability of RRDtool to perform tem- 
poral interpolation (data updates at irregular 
intervals) and conversion of counters to rates. 
Leverages the graphing capabilities of RRD- 
tool. Graphs relevant to aberrant behavior 
detection can be generated using the existing 
syntax of RRDtool. 

e Leverages client software designed to run with 

RRDtool (1.e., Cricket). 


Architecture 


On disk, the round robin database (RRD) is orga- 
nized into sequential sections, round robin archives 
(RRA). Within each RRA is a section for each of the 
data sources (variables) stored in this RRD. Each RRA 
is defined by a consolidation function which maps pri- 
mary data points (PDP) to consolidated data points 
(CDP). At another level, an RRA is just an array of 
data values that is updated in sequence according to 
some function at some fixed time interval. 
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On its face, the aberrant behavior detection algo- 
rithm needs at least two arrays, one to store the fore- 
cast values corresponding to each primary data point, 
and a second to store the predicted deviation corre- 
sponding to each PDP. As implemented, the seasonal 
coefficients and deviations that are used to calculate 
the forecast and predicted deviations are stored in a 
second pair of RRAs. These arrays have length equal 
to the seasonal period and are updated once for each 
PDP. Failures are tracked by a fifth RRA, which deter- 
mines violations and failures on each update. 


The intercept and slope coefficients required for 
the forecast are updated for every primary data point 
and are unique for each data source. As only the most 
recent value of each is required (see “The Holt-Win- 
ters Forecasting Algorithm’’), these parameters are 
stored in a temporary buffer in the header allocated for 
each RRA-data source combination in the RRD (the 
CDP buffer). This buffer is flushed back to disk on 
every call to RRD update. 


The adaptation parameters are the same for all 
data sources within that RRA. They are stored in the 
RRA parameter buffer, which is read only during RRD 
update. 


Therefore, implementation of the aberrant behav- 
lor algorithm adds five new ‘consolidation functions’ 
to RRDtool 
HWPREDICT: an array of forecasts computed by 

the Holt-Winters algorithm, one for each PDP. 

SEASONAL: an array of seasonal coefficients with 
length equal to the seasonal period. For each 
PDP, the seasonal coefficient that matches the 
index in the seasonal cycle is updated. 

DEVPREDICT: an array of deviation predictions. 
Essentially, DEVPREDICT copies values from 
the DEVSEASONAL array to preserve a his- 
tory; it does no processing of its own. 

DEVSEASONAL: an array of seasonal deviations. 
For each PDP, the seasonal deviation that 
matches the index in the seasonal cycle is 
updated. 

FAILURES: an array of boolean indicators, a 1 indi- 
cating a failure. The CDP buffer stores each 
value within the window. Each update removes 
the oldest value from this buffer and inserts the 
new observation. On each update, the number 
of violations is recomputed. The maximum 
window length enforced by this buffer is 28 
time points. 

Usage 
This section illustrates how to use the aberrant 

behavior detection algorithm in RRDtool through an 
example. The monitoring target will be a router inter- 
face on a link between two data centers in the WebTV 
production service network. The variable will be the 
outgoing bandwidth rate (in Mbps). Bandwidth usage 
follows a daily cycle and SNMP is polled at five 
minute intervals. 
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Creating a RRD file 


The first step is to create a RRD for this target 
with aberrant behavior detection enabled. In order to 
simplify the creation for the novice user, in addition to 
supporting explicit creation the HWPREDICT, SEA- 
SONAL, DEVPREDICT, DEVSEASONAL, = and 
FAILURES RRAs, the RRDtool create command sup- 
ports implicit creation of the other four when 
HWPREDICT is specified alone. To take advantage of 
this, use the following syntax: 


RRA:HWPREDICT:<row count?: 
<alpha>:<beta>:<period> 


Where: 

row count is the number of forecasts to store before 
wrap-around; this number must be longer than 
the seasonal period. This value will also be the 
RRA row count for DEVPREDICT RRA. 

alpha is the intercept adaptation parameter, which 
must fall between 0 and 1. The same value will 
be also be used for gamma. 

beta is the slope adaptation parameter, again 
between 0 and 1. 

period is the number of primary data points in the 
seasonal period. This value will be the RRA 
row count for the SEASONAL and DEVSEA- 
SONAL RRAs. 

Using this implicit creation option creates the FAIL- 

URES RRA with a default window length of 9 and a 

default threshold value of 7. The default row count of 

the FAILURES RRA is one period. 


Here is an example of the create command, using 
this syntax: 


rrdtool create monitor.rrd -s 300 \ 


DS sa rOutOctrets + COUNTER? 1800%07429496/295 -\ 


RRA AVERAGE 30657 122016" 4 
RRA: HWPREDICT:1440:0.1:0.0035:288 


After creation parameters can be changed using the 
tune command. RRDtool supports several new tune 
flags: 

--alpha --beta --gamma 

--window-length --failure-threshold 

--deltapos --deltaneg 


Each of these flags takes a single argument that corre- 
sponds to parameters discussed in section “‘Choosing 
Model Parameters.””» The gamma flag will reset the 
adaptation parameter for both the SEASONAL and 
DEVSEASONAL RRAs (setting both to the same 
value). Both deltapos and deltaneg set the scale param- 
eter for the upper and lower confidence band respec- 
tively, the default value for both is 2. 


For example, suppose the technician is unhappy 
with the default window length and threshold for the 
FAILURES RRA implicitly created by the previous 
command. Issue the command: 

rrdtool: cune MOnttorerrd 
--window-length 5 \ 
--failure-threshold 3 
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The remainder of the example uses the default window 
length of 9 and the default threshold of 7. 


Other options of the create command, including 
syntax and details of the explicit creation of the new 
RRAs, are discussed in a detailed implementation doc- 
ument [4] and the RRDtool manual [6]. 


Detecting Aberrant Behavior 


The aberrant behavior detection algorithm 
requires nothing unusual for the RRDtool update com- 
mand; the collection mechanism (i.e., Cricket invok- 
ing SNMP) will run normally. Now suppose some 
time has passed and the network technician 1s monitor- 
ing outgoing bandwidth at the router interface. He can 
view a graph of daily activity, including confidence 


bands and any failures, with the command in Listing 
1, 


TICK is a new graphing option in RRDtool. For 
every non-zero value in the DEF or CDEF argument, 
it plots a tick mark. The length of the mark (line) is 


rrdtool graph example.gif \ 
DEF: obs=monitor.rrd:ifOutOctets: AVERAGE 


\ 
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specified by the third argument (after the color code) 
as a decimal percentage of the y-axis. 1.0 is 100% of 
the length of the y-axis, so the tick mark becomes a 
vertical line on the graph. 


Figure 1 is an example of this daily graph gener- 
ated on Wed, May 31, 2000 for the router target 
described previously. The thin lines are the confi- 
dence bands and the vertical bars represent failures 
(actually multiple failures in sequence — once the 
observed value strays outside the confidence bands it 
remains outside the bands for roughly a two hour 
period in both cases). The TICK graph option gener- 
ates the bars from the FAILURES RRA. 


The graph suggests that bandwidth on this outgo- 
ing link is increasing faster than expected by the 
model (past history). It is up to the network technician 
to decide if this represents aberrant behavior of inter- 
est. One approach the technician might take is to view 
the time series for this router interfiace over a longer 
time period. 


DEF: pred=monitor.rrd:ifOutOctets:HWPREDICT \ 

DEF: dev=monitor.rrd:ifOutOctets:DEVPREDICT \ 

DEF: fail=monitor.rrd:ifOutOctets:FAILURES \ 
TICKitail#frfra0: 1.0: "Failures Average bits out" ’\ 
CDEF:scaledobs=obs,8,* \ 

CDEF:upper=pred.,deve2, 0b A“ 

CDEF:lower=pred,dev,2.%5.~ \ 

CDEF:scaledupper=upper,8,* \ 

CDEF:scaledlower=lower,8,* \ 
LINE2:scaledobs#0000ff:"Average bits out" \ 
LINE1:scaledupper#ff0000:"Upper Bound Average bits out" \ 
LINE1:scaledlower#ff£0000:"Lower Bound Average bits out" 


Listing 1: Graph generation command. 
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Figure 1: Observed bandwidth with Confidence Bounds for May 31 
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With hindsight, it is easy to demonstrate some- 
thing unusual is going on and the aberrant behavior 
detection model 1s catching it in real time. Figure 2 is 
the time series for the week and half period from May 
24, 2000 to June 2, 2000. It is clear that Wed, May 31, 
1s unusual. Bandwidth increases in two steps: once in 
the early morning and again in the early afternoon. In 
this case, the dip to 0 in the early morning hour and 
the subsequent jump can be attributed to a scheduled 
downtime for the service network. Perhaps the 
remainder of bandwidth activity on Wed has the same 
cause, in which case aberrant behavior detected is a 
false positive in the eyes of the network technician. 


Initialization 


As alluded to in the previous initialization sec- 
tion, the implementation is designed to use bootstrap 
initialization. The intercept coefficient is initialized to 
the first observed value. The slope is initialized to 0, 
predicated on the assumption the linear trend over 
time is close to 0. If this is not the case, the time 
required for the Holt-Winters algorithm to gravitate 
away from 0 will depend on the seasonal adaptation 
parameter, gamma. During the first seasonal cycle of 
observed values, seasonal coefficients are initialized. 
During the second seasonal cycle of observed values, 
seasonal deviations are initialized. Unknown values 
during the first two seasonal cycles can complicate 
initialization. Basically, the implementation initializes 
any coefficients it can at the earliest opportunity; refer 
to the detailed implementation document [4]. 


Enhancements to Cricket 


Cricket is a front-end to RRDtool [1, 2]. Cricket 
manages multiple time series via RRDtool in a target 
configuration hierarchy. The configuration hierarchy 
(or config tree) is a flexible approach to grouping tar- 
gets with common time series variables, graphing 
characteristics, and other properties. The Cricket col- 
lector provides built-in and extensible mechanisms for 
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gathering data and feeding it to RRDtool. The collec- 
tor manages SNMP calls and reads application event 
logs. The Cricket grapher generates time series graphs 
using the capabilities of RRDtool in real-time and 
serves them up as web pages. The graphs are orga- 
nized via directory pages generated to match the con- 
fig tree. 


Monitoring 


RRDtool has no mechanism for raising alerts, 
while Cricket does. Cricket provides several types of 
monitor-thresholds, which are defined in the config 
tree in a target dictionary section. Each monitor-thresh- 
old entry can contain multiple monitors. The basic 
functionality of a Cricket monitor-threshold is to fetch 
the most recent value from one of the RRAs of the tar- 
get RRD file, check some criteria, and take some user- 
defined action if the criteria fails. 


For efficiency and simplicity, Cricket 1.1 
includes a new type of monitor-threshold specific for 
aberrant behavior detection. This monitor, failures, 
joins the existing Cricket monitors relation, value, and 
hunt. The general Cricket 1.1 monitor-threshold syn- 
tax permits a comma-delimited list of monitors. The 
syntax of each monitor 1s: 


(data. Source? :?<monitor type: 
Cmonwtor args? Caction? ?Caction args? 


The failures monitor does not take any arguments. For 
example, to send an SNMP trap whenever a failure is 
recorded for the ifOutOctets data source used in the 
example, the network technician adds the following 
entry to the appropriate target dictionary section: 


my -monitor-threshold 
"TfOutOctets: failures: : SNMP" 


Note that currently in Cricket 1.1, SNMP actions do 
not require any arguments, but the tag trap-address 
must be defined in the target dictionary. This may 
change in the future as Cricket 1.1 is still under devel- 
opment. 





I 

’ i 

; 3 a $ 3 3 
2 Hoss Baot} Biahi peu ton hes emsssepraras 
seated ode Bot le io 
monk a oe : ==}. +—? pe mnths pa aati 
av of oe ‘ ~ a : 
a = 5 


Lis Ld best 






AENILIG 252i F WHlLdsy 






Figure 2: Router Interface bandwidth May 24-June 2 
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The failures monitor searches for a FAILURES 
RRA in the RRD file for the specified data source and 
if successful fetches the most recent value. If this 
value is 1, indicating a failure, it triggers the specified 
action. Through this mechanism, a network technician 
can easily be notified when the algorithm identifies 
something of interest. 


Note that while the aberrant behavior detection 
RRAs, if created, apply to all data sources in the RRD 
file, the monitoring mechanism can be enabled for 
specific data sources or none at all. 


HTML Navigation Links 


Given the complexity of the RRDtool graph 
command, the Cricket 1.1 implementation provides a 
simple mechanism to view graphs relevant to aberrant 
behavior detection. 


Cricket generates HTML navigation links to 
graphs using the new RRAs if it detects the string 
‘HoltWinters’ in the name of the view. The view must 
consist of a single data source that is not multi-target. 
It verifies these restrictions before enabling the navi- 
gation links. There are three navigation links added: 
Confidence Bounds: displays the target data source 

using the Hourly Time Range with upper and 

lower confidence bands. The confidence band 
scaling factor (delta) cannot be obtained 
directly from the RRD file. Specify this factor 
with the confidence-band-scale tag in the graph 
dictionary. The default value (if the tag is omit- 

ted) is 2. 

Failures: displays the target data source with 
prospective failures marked with vertical yel- 
low lines (or yellow bars for failures in 
sequence) using the Hourly Time Range. 

Confidence Bounds and Failures: the combination 
of the both of the preceding graphs. 


Conclusion 


There is a need to meet the second challenge of 
networking monitoring: automatic aberrant behavior 
detection. The model and software outlined here are a 
solid approach to the problem, working within the 
architecture of the existing open-source solutions 
RRDtool and Cricket. There is ample room for future 
work, especially in a solution that exploits not only the 
past history of a service network variable, but the rela- 
tionships between service network variables. 


Software Availability 


The RRDtool implementation is available as a 
patch to the current release of RRDtool at 
http://cricket.sourceforge.net/aberrant. This web site 
also includes the more detailed reference document [4] 
on the implementation in RRDtool. The Cricket 
enhancements are part of Cricket 1.1, available at 
http://cricket.sourceforge.net/. 
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ABSTRACT 


When faced with the many problems that arise in a complex of heterogeneous networked 
workstations, systems administrators often resort to coding scripts to monitor and problem-solve, 
scripts that they then schedule via cron. PIKT is a new and innovative approach to monitor 
scripting and managing system configurations. PIKT consists of an embedded scripting language 
with unique labor-saving features, a sophisticated script and system configuration file 
preprocessor, a scheduler, an installer, and other useful tools. More than just a systems monitor, 
PIKT is also a cross-categorical toolkit for configuring systems, organizing system security, 
formatting documents, assisting command-line work, and performing other common systems 


administration tasks. 


The Problem, A Solution 


Sysadmins have long wrestled with the task of 
writing generalized scripts to monitor systems and 
deal with recurring problem situations. As conven- 
tionally practiced, this approach has numerous disad- 
vantages: it is hard to account for diversity across 
machines and operating systems; operations are fragile 
and error-prone; scripts for handling simple tasks are 
difficult to code, or are hardly worth the effort to 
maintain; scheduling and managing scripts are time- 
consuming and repetitive; setup is inflexible; activity 
and error logging is rudimentary or nonexistent; and 
the whole mass of scripts and configuration files is 
nearly impossible to keep track of or even compre- 
hend. 


PIKT attempts to solve some of the problems 
observed in more traditional methods of monitor 
scripting and managing system configurations. PIKT 
is an embedded scripting language and accompanying 
script interpreter. PIKT is also a sophisticated script 
and system configuration file preprocessor for use 
with the Pikt scripting language or any other scripting 
language of your choice. Finally, PIKT is a cross-plat- 
form, centrally run script scheduler (like cron), cus- 
tomizing installer (like rdist), command shell enhance- 
ment, and total script and configuration file manage- 
ment facility. PIKT’s primary purpose is to monitor 
systems, report problems, and fix those problems 
whenever possible, but its flexibility lends itself to 
quite a few other uses as well. 


Overview 


In the usual PIKT configuration, you manage the 
monitored client (‘‘slave’”) machines from a central 
(“‘master’’) control machine. On the central control 
machine, there are eight controlling config files: sys- 
tems.cfg, defines.cfg, macros.cfg, alerts.cfg, alarms. 
cfg, objects.cfg, programs.cfg, files.cfg. They define 
your entire setup. You invoke the overseeing manage- 
ment utility, piktc (for “‘pikt control’’) to preprocess 
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those files, to install client target files, and to perform 
other management functions, such as stopping and 
restarting daemons. 


Two daemons run on each client, piktc_svc and 
piktd. piktc_svc listens for and responds to piktc 
requests. piktd launches Pikt scripts at specified inter- 
vals. 


On all clients, piktd wakes up every minute to 
check if one or more groups of scripts, also known as 
“‘alarms’’, are due to run that minute. Alarm scripts 
are grouped together as “‘alerts’’. Alerts run at specific 
intervals, e.g., hourly, once daily, once weekly, etc. At 
the appropriate time, piktd summons the pikt inter- 
preter to run the Pikt scripts for that alarm group. You 
can also run Pikt scripts manually at the command 
line, but usually they are invoked by piktd. 


pikt is the Pikt scripting language interpreter. 
Individual Pikt scripts usually monitor just one aspect 
of a system. You can monitor a single object, or col- 
lections of things listed in the object files, for exam- 
ple: system processes, disks, devices, lists, etc. 


Each Pikt alarm script gets its input from pro- 
cesses, files, or log files. For log files, only new log 
entries since the last alarm run are considered. 


A typical alarm consists of a sequence of logical 
tests. If the current input line satisfies one or more 
conditions, actions may or may not be triggered. Con- 
ditions might also refer to data in the previous input 
line, to data for this line remembered from the prior 
alarm run, even to data coming from outside the cur- 
rent alarm and pikt process. 


Triggered actions might include generating a line 
of e-mail. At the end of the current pikt run, queued 
lines are e-mailed in a single problem report to one or 
more sysadmins. Queued lines might be printed or 
logged, whether to syslog, to this alert’s log file, or to 
some other special log file. Or commands might be 
executed, for example to restart a detected dead sys- 
tem process, to chown a file, or perhaps to page the 
sysadmins. 
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For generating alarm script input, for taking 
action, also for serving as subroutines, you can 
employ auxiliary programs and scripts written in 
other, non-Pikt languages. 


All external commands are logged for debugging 
and auditing purposes. If your alarm script makes ref- 
erence to data from the prior alarm run, current data is 
stored in history files for looking up next time. And, 
very importantly, all errors are logged (including 
errors generated by invoked scripts written in other 
languages), giving you a complete audit trail when 
things go wrong. 

You may also employ PIKT to manage system 
configuration files, such as inetd.conf, syslog.conf, 
sudoers, etc. It becomes much easier, for example, to 
enforce consistent access rights across your many sys- 
tems. 


The PIKT binaries are written using a combina- 
tion of C, lex (flex), and yacc (bison). Most of the 
sample scripts are written in the Pikt script language, 
although several auxiliary Perl, expect, and shell 
scripts are also provided. 


Configuration 


Config Files 


Every config file is a sequence of stanzas. A 
stanza consists of a stanza identifier, in the first col- 
umn, then the stanza body, either on the same line or 
in multiple lines following. The stanza body must be 
indented, using spaces, tabs, or the #indent preproces- 
sor directive. 


Almost without exception, and aside from the 
above simple rules, PIKT is indifferent to script and 
config file layout. In other words, spacing and line 
breaks really don’t matter, and you may lay out your 
config files in any style that pleases you. 


PIKT comments are like those in C+-+, that is, // 
and/* */, 


systems.cfg 1s where you specify host systems, 
host aliases, and host groups. Here is an example: 


LIAS TIA TIA TATAS TITAS ALAA TLE LA LTD 
// 
// PIKT systems configuration file 


VIAL LT ATTATTATATSATAA STADIA TAT ALTA TAT 


Solaris 
hosts moscow athens2 
berlin milan. ondon 
paris paris4 paris5 
ins 
hosts murmansk firenze 
Milan 
aliases bonn rome 
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mailserver 


members moscow paris 


defines.cfg specifies a set of “defines” — logical 
switches for including or excluding sections of the 
config files. Here are some example defines: 


debug FALSE 
verbose FALSE 
paranoid TRUE 


macros.cfg specifies a collection of text substitu- 
tions. Macro definitions, but not macro names, may 
include embedded macros. Some examples: 


behead (N) =sed ’1,(N)d’ 


offhours C dhour() 2= 18° |||. 
Prieur OG Gz) 
sysadmins brahms|albeniz|liszt 


In config files, a macro reference is preceded by 
an equal sign, for example: =behead(1), =offhours, 
=sysadmins. 


Macros may also include macro arguments. As 
with simple text-substitution macros, macros-with- 
arguments may reference other macros-with-argu- 
ments, so all manner of macro nesting is allowed. 


In alerts.cfg, you schedule alarm scripts. Here is 
an example alerts.cfg stanza: 


Urgent 

timing Ora ace tk 

drite 5 

Droriry 10 

mailcmd Memadls =s “PIKT \ 
hlLert som -Shost<\ 
Urgent’ =pikturgent" 

lpcmd val Dp -Spiktprinter 

alarms 
SysRebootUrgent 
FsMountsUrgent 


SwapChkUrgent 


The timing parameters follow the usual cron con- 
ventions and then some. One not so usual timing spec 
is random timings (for example, ‘timing 20% * * * *’, 
which says to run the alert on average every five min- 
utes). The random timing spec is especially useful in 
security situations, where you want some _ unpre- 
dictability in your monitoring schedules. Still another 
novel timing spec is “‘drift’”» — how many minutes an 
alert launch may randomly occur before or after a 
specified time — useful when you don’t want alerts to 
“bunch up.” 


As alarm scripts are run, their output is queued. 
At the end of the alert run (an alert is a set of alarms), 
the queued output may be sent as a single e-mail mes- 
sage to one or more sysadmins, or printed out, using 
the commands specified. 
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The alarms.cfg file 1s a series of Pikt scripts or 
alarm definitions. For a more detailed discussion of 
Pikt scripts, see the Scripting Language section below. 


In objects.cfg, you specify system objects to be 
monitored. Object listings can also include data 
parameters. For example: 

UserDirs 

#if kiev2 
/pub/mus_disk_5 
/pub/mus_disk_6 

fHelif kiev0O 


dFendif 


SysProcs 


cron # /etc/init.d/cron start 


The file programs.cfg contains support scripts 
written in other scripting languages, with each pro- 
gram in its own stanza. 


In files.cfg, you can centrally manage system 
configuration files (such as inetd.conf, motd, and so 
on), and indeed any text file. files.cfg 1s much like 
programs.cfg, except that it can and should contain 
non-program files and/or programs external to the 
PIKT setup. 


Partial Configurations 


In a complete PIKT setup, you have all eight 
basic configuration files. You might, in addition, have 
#include file spinoffs from those basic eight. 


It is possible to deploy PIKT in a partial configu- 
ration, with subsets of the eight basic config file types. 
systems.cfg is always required, but all the rest are 
optional. 


Here are the most common PIKT setups: 

e piktc as rsh/ssh replacement (no macros or 
defines): systems.cfg only 

e piktc as rsh/ssh replacement (with macros and 
possibly defines): systems.cfg, macros.cfg; and 
optionally defines.cfg 

e piktc as rdist replacement: systems.cfg, files. 
cfg; and optionally programs.cfg, macros.cfg, 
defines.cfg 

ea centrally managed cron replacement: sys- 
tems.cfg, alerts.cfg, alarms.cfg; and optionally 
macros.cfg, defines.cfg 

e system/network monitor (but without system 
files management): all config files except 
files.cfg 

e system/network monitor; rsh/ssh, rdist, cron 
replacements: all config files 


So, you may utilize all that PIKT has to offer, or 
just pick and choose among its many functionalities. 
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Preprocessing 


piktc & piktc_sve 

PIKT is managed through the combined action of 
the interactive control program, piktc (on the central 
master machine only), and the piktc_svc service dae- 
mon (on all slave machines). 


The piktc command options are shown in 
Appendix 1. 


When specifying items, you include items with 
“+”? and exclude with “-”. For example, “+A all” 
includes all alerts. “+A all -A EMERGENCY Info” 
includes all alerts except EMERGENCY and Info. 
Another way to achieve the same effect is with just 
‘“-A EMERGENCY Info” (leaving out the “+A all’’, 
which 1s implicit). 

This sample command checksums (using MD5) 
all files on all user systems except the Linux machines 
and any down systems: 


## piktc -m5v ALL -H nonusersys 
linux downsys 


Preprocessing 


You use piktc to preprocess source configuration 
(*.cfg) files on the master machine, and send the post- 
processing alert (.alt), object (.obj), program, and 
Other files over the network to receiving piktc_svc 
daemons for installation on the slave systems. Prepro- 
cessing entails: 

¢ stripping out meta-comments (comments of the 
form // or /* */) 

e #include’ing auxiliary files (e.g., a list of Unix 
command macros) 

e using #if <os|host|hostgroup> #endif preproces- 
sor directives, filtering through lines pertaining 
only to the current client (e.g., #if solaris) 

e using #ifdef <define> #endifdef preprocessor 
directives, for including/excluding portions of 
the text (e.g., #ifdef debug) 

e making macro substitutions (e.g., substituting a 
Unix command path, with command options, 
appropriate to the current client) 

e performing an across-the-board syntax check 


Note that, in addition to the Pikt script and con- 
fig files, it is possible to use meta-comments, 
#include’s, #if’s, #ifdef’s, and macros in managed sys- 
tem configuration files and scripts written in other lan- 
guages (e.g., Perl [11], Python [6], AWK [2]). Note, 
too, that scripts may rewrite config #include files, rais- 
ing interesting possibilities for maintaining dynamic 
system configuration files. 

Preprocessor Directives 


You can customize config files by means of the 
#if, #elif, #else, and #endif preprocessing directives. 
The format is 
#Fif <machine class> 


<lanes? 
d4telif <machine class> 
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<lines> 
tHelse 

<lines? 
ftendif 


where <machine class> can be a series of host names, 
host aliases, or host groups, separated by the |, &, or ! 
set operators. | indicates set union, & set conjunction, 
and ! set negation. You can also use parentheses, ( and 
), In the class speci fications. 


Akin to #1f, a second class of preprocessor direc- 
tives consists of: #ifdef, #ifndef, #elifdef, #elifndef, 
#Helsedef, #endifdef, #define, and #undefine. The for- 
mat is 

fHifdef <define> 
<lines? 
ftelifdef <define> 
<lines> 
ttelsedef 
<lines> 
fHendifdef 


where <define> is an identifier representing a type of 
logical switch that is either defined (true) or undefined 
(false). 


Logical defines are set (to TRUE) or unset (to 
FALSE) in any of three ways: (a) in the file 
defines.cfg; (b) in any config file, except systems.cfg 
or defines.cfg, by means of the #define and #undefine 
directives; or (c) at the command line, by means of 
either the +D or -D switches. 


Observe that you can set and unset defines on a 
per-machine basis in the defines.cfg file, for example 


tif dbserver 


paranoid TRUE 
Helse 

paranoid FALSE 
#endif 


as well as nest #ifdef’s within #if’s, and vice-versa, 
throughout the config files. 


A config file can incorporate one or more other 
files by means of the #include directive. Included files 
may themselves include other files, but only of the 
same basic configuration type (macro files include 
macro files, for example). Here is an example 
#include directive: 


#tHinclude <security_alarms.cfg> 


Includes are especially useful for compartmental- 
izing across different systems administrators (where 
each has his/her own sub-config file), and across func- 
tions (e.g., security alarms in one file, network alarms 
in another), and for including files contributed by out- 
siders. Includes are also good for quarantining infor- 
mation particular to different operating systems. 


There are other preprocessing directives, but the 
ones described above are the most common. 
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Scripting Language 
Script Outline 


The general outline of a Pikt script 1s: 


<script name> 


io 
status active| 
inactive 
level emergency | 
urgent | 
eriiacal| 
task "Ctext? 
cnt. proc "<process>" 
file wet ep" 
loprite. “"Clogrileo”™ 
filter "<process2" 
seps "Cchar(s)>" 
dat <var?> <spec? 
keys var? ese | 
begin 
<statement> 
rule 
<statement> 
end 
<statement> 


Init Section 


In the init section, you lay the basis for subse- 
quent script actions. 


The alarm is given one of eight severity levels, 
analogous to syslog’s severity levels. 


The primary alarm input is the output of a pro- 
cess, the full contents of a text file, or logfile updates 
(new info since the previous alarm run). If a process, it 
can be any system process (including multiple pro- 
cesses tied together by pipes) yielding text output. Pikt 
does not deal with binary input. Input may also be 
passed through an optional filter. 

One or more dat statements map input data to 
variables. The dat statement takes one of three forms: 

dat Cyar> x 


dat “var? xy 
dat "<regexp>" 


[ordinal] 
[columnar] 


For ordinal input, “seps”’ specifies a field sepa- 
rator (or separators) other than the default (whites- 
pace). 

Concluding the init section, the optional keys 
line lists variables used as database lookup keys when 
referring to history values (values stored from previ- 
ous script runs). 


Begin, End, and Rule Sections 


Next come action statements, grouped into 
begin, end, and rule sections. 
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The heart of a Pikt script is the main processing 
loop: A line of input (from a proc, file, or logfile) is 
read in, then acted upon, the next line is read in and 
acted upon, and so on until the input is exhausted. 
Before input processing, you might have a begin sec- 
tion, to initialize some variables or take some other 
preliminary actions. You might also have an end sec- 
tion for input processing followup. (You can achieve 
additional data processing loops within a Pikt script 
using a combination of #fopen(), #popen(), #read(), 
and #fclose(), and/or #pclose().) In other words: 


begin [optional] 
<statement> 
<statement> 
fwhile theres input] [optional] 
rule 
<statement> 
(statement? 
[endwhile] 
end [optional] 
<statement> 
<statement> 


The input processing loop consists of one or 
more rule sections. A rule section usually groups 
together program statements pertaining to a single 
attribute of the current input line. Strictly speaking, 
there is never a need to break up the set of input pro- 
cessing statements into separate rule sections, but 
doing so helps clarify program logic. 


Data Types 


Pikt has three basic data types: strings, numbers, 
and file handles (or proc handles). 


Pikt supports both “associative” (string-indexed) 
and numeric (numerically-indexed) arrays. Array 
indices are computable (e.g., a concatenation, or the 
sum of two functions). Numerical arrays are (for now) 
limited to at most three dimensions. Note that, unlike 
with many other languages, Pikt array indices start at 
1, not 0. (In Pikt, generally speaking, all indexing, in 
whatever context, begins with 1.) 


Variables come in three different time forms. 
““$” and ‘“‘#” as variable prefixes refer to current val- 
ues (strings and numbers respectively). “@” (e.g., 
@uid) signifies the value for this vanable for the pre- 
ceding input line. “%” (e.g., Yousage) signifies the 
value for this variable during the previous script run. 

So, in Pikt, there is no need to save input data 
values from one line to the next. Values from the pre- 
vious input loop are stored automatically for you. The 
same is true with so-called “history variables.’’ Pikt 
stores values in a data file for recall the next time the 
alarm script runs. If a value is tied to a particular input 
data variable (specified in a dat statement) and a par- 
ticular line of input, Pikt does a keyword lookup 
(specified in a keys statement) to find the appropriate 
data value. 
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Other Language Features 


In general, every Pikt object serves a semantic 
purpose. Hence, and for example, parentheses are not 
required around an if condition or the arguments to a 
for statement. Nor are semicolons or end-of-lines 
required to signal the end of a program statement. 


Pikt provides the usual operators, and a few not 
so usual. They mostly follow the Perl and AWK mod- 
els. 


Pikt offers a wide variety of built-in functions. 
An unusual feature of Pikt functions is that they are 
data-typed: their return value type is signified by 
either the “$” prefix (for string; e.g., $trim()) or “#” 
prefix (for number; e.g., #median()). Pikt does not cur- 
rently support user-definable functions, although you 
can write pseudo functions using macros-with-argu- 
ments to achieve much the same effect. 


Pikt comes with a panoply of flow control struc- 
tures, most usual, and a few not so usual (e.g., ‘again’, 
for repeating the current rule; leave’, for leaving the 
current rule). Every Pikt statement begins with a key- 
word (e.g., ‘set’, ‘if’, etc.). Statement blocks are indi- 
cated by a keyword-keyword combination, for exam- 
ple, if-endif, for-endfor. 

Pikt uses AWK and GNU RX-style regular 
expressions. 


Several Monitoring Examples 


It should be emphasized that the examples fol- 
lowing are not an intrinsic part of PIKT. They are 
solutions that you might implement, not that you are 
forced to adopt. 


Case Study 1: IdleUserSession 


IdleUserSession is a short Pikt script to kill aban- 
doned user sessions. Listing 1 is the source version on 
the master control machine as it would appear in the 
alarins.cfg file. 

We have decided that this needs to be run every 
other hour or so, so we group it with other “critical” 
alerts in the alerts.cfg file: 


CristieaL 
timing 30702092" A> = 
Grate 5) 
dif moscow | munich 
Priority 10 
dFelse 
PLLOrITy 0 
dFendif 
mailcmd ‘=matix Ss -PIKT \ 
Alert Sones 
=pikthostname: \ 
Gritieal™ 
=Dikterird ca" 
alarms 
IdleUserSession 
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We would install this alarm, along with the other 
alarms in the Critical alerts group, with the command 


# pikte -iv +A Critical -H downsys 


processing madrid2... 
ds bade ner a els ain. 
Critical.alt installed 


We have defined macro command paths in 
macros.cfg like so: 


+#if solaris 


eine iL /usr/bin/kill 
awk /usr/bin/nawk 
fFendif 


If the current client were defined as a solaris sys- 
tem in the PIKT systems.cfg file, the piktc preproces- 
sor installs this script on the client (in the Critical.alt 
file) with the macros resolving to the appropriate 
solaris command paths, as in Listing 2, for example. 


Note how macro substitutions have inserted the 
appropriate paths for the w, nawk, ps, and kill com- 
mands. If this were for one of the other supported 
operating systems, different paths would be inserted. 


You no longer have to concern yourself with 
specifying the correct path for this or that command in 
your scripts, either by maintaining separate script ver- 
sions or by inserting per-OS case statements into your 


IdleUserSession 
eat 
Status active 
eve erait ea) 
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code. Simply define the path once and for all in the 
macros.cfg file, then use the =nawk macro (for exam- 
ple) ever after in all of your scripts (including scripts 
written in other languages, such as Perl, AWK, etc.). 
PIKT will automatically substitute the correct version 
for you. 


Input data results from the command ‘‘=w’’, 1.e., 
‘‘/usr/bin/w’’. Here 1s a sample input line: 


Bach pts/4 29Jun98 Sdays 3:25-2 zsh 


We pass this input along to nawk with the 
instructions: match lines showing idle time in days; 
transform, for example, “‘pts/4’” into “pts\V4’’; output 
just the first and second fields. 


Pikt maps the nawk output “bach ptsV4”’, setting 
$user to the first field and $tty to the second. 


This alarm has but one rule: We exec a kill com- 
mand to terminate the idle session in question. (The 
exec is automatically logged for auditing and debug- 
ging purposes.) 

You could, if you want, add rules to kill root ses- 
sions only, or to kill after midnight and on weekends, 
or if certain other conditions are met. Instead of 
killing, you could send e-mail alerts to the system 
administrators, who could then decide if manual ses- 
sion kills are required. 


Case Study 2: FileStatChk 


One thing you would certainly want to monitor 1s 
the state of essential system files: Have they 


task: “Terminate idle user. sessions.” 
input proc. "Sw: || ‘snawk 7 [i 9iiday/ (gsub (Vet ANA ak 


Prine tol Vase ne 

dat Susser. a 

dat Stty 2 
rule 

=execwait "=kill ‘=ps -ef | =nawk ’/Suser.+$tty/ {print \$2}°‘" 

Listing 1: IdleUserSession (source version). 
IdleUserSession 

init 


Status active 
level -eritical 


task "Terminate idle user sessions." 
input: proe-"/usr/bin/w.\|, 7 dsr/ ban/nawk */)[1-9 |-day/ “\ 
osu NN) ee NON ep ra bo. aM CO raza 


dat Suser 1 
dat stiy 2 
rule 


exec wait "/usr/bin/kill ‘/usr/bin/ps -ef | \ 
fusr/bin/nawk “/Suséer.totty/. {print \o2) es 


Listing 2: IdleUserSession (target version). 
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disappeared? Do they have the right ownerships and 
permissions? 

We start by listing those files, together with their 
desired attributes, in objects.cfg (see Listing 3). 


If we had adjusted the files list for the moscow 
system only, we would refresh the SysFiles objects set 
on that system with the command: 


SysFiles 

#if linux 
/etc/group -rw-r--r-- 
/etc/passwd -rw-r--r-- 


ffendif // linux 


Li eeal- Sstutt 


tif moscow 
/etc/mail/classalias -rw-r--r-- 


endif 
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## pikte -iv +O SysFiles +H moscow 
processing moscow... 
instadlm or ot il er(s))4 ee 
SysFiles.obj installed 
We could refresh all objects files on all active 
systems with the command 


## piktc -iv +O all -H downsys 


644 root root 
644 root root 
644 root other 


Listing 3: SysFiles. 


Pi teStatchk 


ae hit te 
Status active 
Keved--eritieal 


task "Detect critical file access deviations on system files." 


input file "=sysfiles_obj" 
dat) -Sitil 1 

dat Sprm 
dat Smod 
dat Sown 
dat “Sorp 
keys $fil 


in i W bb 


rule 
doe oo et ota 


Output Madi "Stil nor found! 


next 
endif 


rule 


do #split(Slist, Scommand("=lld $fil"), ™") 


rule 
defo ste) sie: Sp rl 


=execwait "=chmod Smod S$fil" 


=outputmai |. "Stil “permissions Slist li} -are-wrong” 
Sitidetined(Wilaste ll) 3" Gerexdase (Mi): aay) \ 


"changed to Sprm" 
endif 


[similar rules: ftollow| 


Listing 4: FileStatChk 
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It should be clear by now that the file 
/etc/mail/classalias would appear in moscow’s Sys- 
Files.obj file and in no other system’s. 


Listing 4 is a script to enforce those file 
attributes. 


For the first input line, “‘/etc/group”’ would be 
assigned to $fil, “‘-rw-r--r--” to $prm, ‘‘644” to $mod, 
and so on. 


In the first rule, if the file fails the existence test, 
that gets reported, and we move on to the next input 
line. 


In the next rule, we take the output of the ‘ls -I’ 
command and #split() and assign the component parts 
to the $list[] array. 


In the third rule, if the actual file permissions, 
$list{1], do not equal the desired permissions, $prm, 
we fix and possibly report this. 


The doexec define lets us control whether actions 
are exec’ed else a report of intent is e-mailed only. If 
this is a new PIKT installation, we might want to see 
what PIKT would do before committing PIKT to actu- 
ally doing it. We could handle the conditionality this 
way: 

ifdef doexec 
exec wait "=chmod $mod $fil" 
ftelsedef 


rule 


output, Jog “=swapchk los" = Sinline() 
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output mail. "=chmod -sSmod > Sfil" 
fFendifdef 


But defining the following macro 


execwait 
fHifdef doexec 
exec wait 
ftelsedef 
output mail 
fFendifdef 


in macros.cfg is more elegant, because now we can 
more succinctly write 


=execwait "=chmod $mod S$fil" 


and either “exec wait” or “output mail”’ will be pre- 
processed in depending on how we defined doexec 
earlier. 


In most circumstances, we simply want the file 
permissions fixed and don’t need to be told about it. 
Sometimes, however, we want a full report of all that 
PIKT is doing. We control this by setting, in 
defines.cfg, the define verbose to be TRUE or FALSE. 
By defining the outputmail macro in macros.cfg as 


outputmail 
ttHifdef verbose 
output mail 
fHelsedef 
output: log "“/dev/nuli" 
fHendifdef 


end // only report if use is very high and increased by at 
jf veast-5% since last.time. (hence don’t “report when 


// swap use is high but declining) 


set fuse = (#blksum-#fresum) /#blksum 


if ( #use >= 80% ) 

&&  ( ( | t+defined(%use) ) 
| ( *%use < 80% ) 
| 


| 
| 
) 


( #use - %use >= 5% ) 


output mail “swap utilization is $text(100*#use,0)%:=newline" 


output mail "swapfile 


for #i=1 #4i<=tinnum() #4it=1 


output mail $line [#i] 
endfor 
output mail =newline 


dev swaplo blocks free" 


output mail $command("=dfk /tmp | =behead(1)") 


=dutop(10, /tmp) 


output mail "contents of /tmp:=newline" 


do t#tpopen(LL, "=] 7] /tmp™, Wr) 


while #fread(LL) > 0 
Output mail. Srdlin 
endwhile 
do #pclose(LL) 
output mail =newline 
=toptop (20) 
endif 


Listing 5: SwapChk (fragment) 
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we can concise ly wnte 


=outputmail "S$fil permissions 


Pony 


If verbose is set to FALSE, the message is logged to 
/dev/null, that is, just thrown away. 


Note the $if(#defined(%list[1])," (were %list[1]),",",’). 
If we have run this script before, we have a record of 
the actual file permissions the last go-around in 
“%list{1]. PIKT remembers this for us automatically. 
So if #defined(“clist[1]) is true, we report what they 
were, and in any case report what they have been 
changed to — but only if we have set verbose to TRUE. 


Case Study 3: SwapChk 


Another thing we monitor 1s if systems run out of 
swap space. For that purpose, we use the SwapChk 
script, a portion of which is shown in Listing 5. 


The input for this script comes from input proc 
"=swap -| | =behead(1)". The last rule above logs all 
input. This might come in handy some day if we need 
data to justify purchase of additional RAM. 
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At the end of all input, we compute #use as a 
percentage. If #use is equal or greater than 80%, or if 
“use 1s not defined (because this is the first alarm run, 
say), or if Youse was less than 80% previously, or #use 
has gone up by at least 5% over the previous “ouse, we 
format a report and send it off as alert mail. Listing 6 
is a sample report. 


PIKT has assembled for us automatically all the 
diagnostic information we need to assess the situation. 
Moreover, after we have identified user freil as the 
memory hog, we can simply add some extra com- 
ments to the top of this alert e-mail and forward it 
along to freil — demonstrating one advantage of using 
e-mail as PIKT’s primary notification mechanism. 


We could also, at least under certain circum- 
stances or on certain systems, augment swap space on 
the fly by adding the appropriate Pikt exec statements. 


Case Study 4: ProcCountsChk 


Recently, we have faced a crisis where a bug in 
the current version of our Web-based e-mail client has 


PIKT ALERT 
Bhi Aug 17) 21320: £42000 
paris6 
URGENT: 
SwapChk 


Report when swap use is high 


swap utilization is 98%: 


swapfile dev swaplo blocks free 
jdevidsk/eOT0d0st 32,4 16 1003184 24384 
/pub/perf_disk_20/swap - 16. 524272 0 

Swap 803568 757800 45768 95% /tmp 
758376 /tmp/SAS_worka0000420D 

8 /tmp/screens 

240 /tmp/ups_data 


contents of /tmpa 


total 544 

Grwie sara > 2 freil perf 629 Aug 17 21:18 SASsxwork 

drwxr-xr-x 2 TOOL other 69 Aug 16 06:15 screens 

SOWakwW ob larooxt sys 239160 Aug 16 11:12 ups_data 

last pads L7Ol4;. ~“load-averages:: 0.20% -0.23,,. O23 “29520521 

54 processes: 46 sleeping, 3 zombie, 4 stopped, 1 on cpu 

Memory: 128M real, 1576K free, 738M swap in use, /7984K swap free 
PID USERNAME THR: PRE NICE SIZE RES STATE TIME CPU CMD 

16845 freil Ip 35 0 12M 3336K sleep £2) © 922863 

16909 freil 3. O 6432K 1464K sleep 37> 342 -sas 

16969: rook ly 333 O 4872K 2792K sleep OF 00% 2.280% pike 


Listing 6: SwapChk (sample report). 
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ProcCountsChk 
are 
Status active 
level emergency 
task "Report unusually high counts of per-user procs." 
// note: a defunct process might show an empty comm field 
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// below, so we pipe the ps output through the awk filter 

input proc "=ps -eo user,comm | =behead(1) | =awk ’NF==2’ li 
=sort | =uniq -c" 

dat #count 1 

dat Suser 2 

dat: -Sproe. -3 


begin 


if #fopen(PROCCOUNTS, "=proccounts_obj", 


while #read(PROCCOUNTS) > 0 
if #split(Srdlin) == 5 


set #1lgcent[$1] = #val($2) 
set #alcnt([$1] = #val ($3) 
set #pgcnt[$1] = #val($4) 
set #klcnt [$1] = #val($5) 


// else send an error message? 


// read in-process: and threshold data. from 7 


ects file 


wrt) Ferr() 


// log thresholds 
// alert thresholds 
// page thresholds 
ji kel “ehresholads 


ee 
endwhile 
do #£close(PROCCOUNTS) 
else 
output maid. "Cant open —proccountsob;- for reading!" 
quit 
rat 
rule 
foreach #keys(Spr, #lgcent) 
if proc." Wh spron § Wigs Me mot -vaqe SO rthaes | Ne 
// works as) a default 
if #Agent [Spr] && f#eount >= #1gent [Spr] 
// for gathering diagnostic stats 
output. log "“=proccounts los” .Siniine 
ry 
if #alcnt[Spr] && #count >= #alent [Spr] 
output mail Sinline 
if Sproc eq "“imapd" // special case 
=archive_mail file(Suser, #true()) 
Ta 
aL 
if #pgcnt[Spr] && #count >= #pgent [Spr] 
exec wait “echo “=pikthostname:. Sinlin™ || \ 
=mailx -s ‘'=pikthostname: Sinlin’ \ 
=pagesysadmins" 
pause 5 
ro 
if #klicnt[Spr] & #eount >= #klicnt [Spr] 
=kill_user_proc(Sproc, $user, #true()) 
Be 
next if next anput bine 
a 
endforeach 


Listing 7: ProcCountsChk 


the imapd, under occasional and mysterious circum- 
stances, spawning instances of itself every second or 
so. For a handful of users, we are seeing occasional 
“ymapd storms” with per-user imapd counts reaching 
into the dozens, hundreds, and sometimes even thou- 
sands! At about the same time, but for different rea- 
sons, we began seeing “ypserv storms’. Not only do 
these storms risk losing user mail files, they also 
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have put into operation to deal with these sorts of 
problems. 


The input proc statement yields input like 


a4 “root! /usr/lib/sendmads 
404 chico imapd 
1 zeppo imapd 
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In the begin section, we read data in from the Instead of foreach #keys($pr, #lgcnt), we could 
ProcCounts.obj file (see Listing 8). have used for $pr in #keys(#Igcnt). These accomplish the 
In the script’s only rule, we check to see if the same purpose but with somewhat different syntax. 
actual per-user process count exceeds the thresholds Variety of expression and keyword synonyms are typi- 
we set in the begin section, also if the threshold is cal of Pikt. Did you notice the use of if ... endif in Case 
non-zero. Studies 2 and 3 as opposed to if... fi in the current case 
ProcCounts 


//-O signifies take no action: 1. signifies always take action 


// 
i 


if 
if 


if 


if 
if 


if 


fey 


proc log alert page kill 
if moscow 

imapd 10 100 1000 100 
endif 
if mailserver 

sendmail 50 100 200 200 
else 

sendmail 5 10 20 40 
endif 
if nisserver 

ypserv Z 5 3 0 
endif 

erack 1 1 1 1 

Sniftit 1 1 i 1 


/]/ wild card should be last in ProcCounts list 


ars 10 20 40 0 
Listing 8: ProcCounts (objects.cfg fragment). 


ka bluse proc (Pr, Uy, M) 


ij Kill off allo iyetancesiof ia gtven process for<a, piven liser 


hi (P).- ts the process mame: (ese. ).-Sproc..or “imapda™) 

jf (Uy) Lethe ser Ce.9%, Suser. or “root") 

// -(M) is whether or not to output mail (e.¢.,. #rrue()) 
set #killcount = 1 ff San tie lave 


while #killcount > 0 
set #killcount = 0 
do #popen(KILL, "“=ps -eo pid,user,comm", "r") 
while #fread(KILL) > 0 
if #split(Sreadline) != 3 
cont 
Ea 
io 6228. CU) 
&G $3:eq (CP) 


fHifdef debug 


outpun soe “=proceounte: lop Sissy oo" 
output lop “"=proccounts log “thy, CU), sotexc cu.) 


dtendifdef 
exec wait "=kill -9 $1" 
set #killcount t= 1 
sae 
endwhile 
do #pclose (KILL) 
endwhile 
if (M) 


GULPUE mai kal ed.alils CU) CP) processes’ 
cal 
Listing 9: kill _user_proc() 
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study? Another example: elif, elsif, elseif are synony- 
mous, and all achieve identical effect. 


If the #lgcnt[] threshold is non-zero and if the 
process count exceeds the #lgcnt[] threshold, we log 
some diagnostic statistics for post-mortem analysis. If 
the process count exceeds #alcnt[], we send alert mail 
reporting that fact. In the case of imapd only, we also 


UserActivity 
qe 
Status “active 
level. e¢rarical 
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backup the user’s mail file by means of the 
=archive_mail_file() macro (not shown). 

If #count exceeds #pgcnt[], we send a short alert 
message to =pagesysadmins, a macro that resolves to 
the sysadmins’ pager numbers. 

Finally, if #count exceeds #kicnt[], we kill off the 


user processes by means of the =kill_user_proc() 
macro (see Listing 9). 


task "Report and/or Jog suspicious after-hours activity." 


iiput prec. = ~hs" 
=wdata 


begin 


exec wait "=touch =useractivity_log" 


#tifdef worried // or paranoid 


/) forced update 


if #false() (7 never “quit this alarm 2% we re 
{} ~moryued) (oe pardnoid) +-monitor 
Ff actayteycat-all-hours 


fFelsedef 

if fhour() >= 8 // 8 AM to midnight only 
dtendifdef 

quit // “don tamoni tor, move: onto mext “alarm 
endif 
rule 

#tifdef worried // or paranoid 

ae dt rue () J} all sisers 
dtelsedef 
df if nonusersys 

af d#true() // all wsers,.-on admin systems 
dF else 

ats Susser eq “root’ <// root only, on user systems 
4 endif 
dtendifdef 


&& ( fHlength(Sidle) == 
| | Sidle = "*[0-9]+5" 


) 


jf idle tine. an minutes. 
// not hours. .or-days 


// escalate notification at higher levels of security 
output log *“=uséractavity log" Sinline 


fHifdef cautious // or worried or paranoid 


output syslog Sinline 
output madd “Sinline 
dtendifdef 
#tifdef worried // or paranoid 
output print Sinline 


dtendifdef 
#tifdef paranoid 
exec wait “echo "=pikthostnames Sinlin’ | =mailx -s \ 
‘=pikthostname: S$inlin’ =pagesysadmins" 
deendifdef 
endif 
Listing 10: UserActivity. 
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Here is a sample alert message: 
PIKT ALERT 


Tie Apr 25. 00e1 702" 2000 
MOSCOW 


URGENT: 
ProcCountsChk 

Report unusually high counts \ 
of per-user procs. 

404 chico imapd 

saved user mail file as 
/var/mail/arc/chico. 956639822 

killed all chico imapd \ 
processes 


(We still don’t have an understanding of these prob- 


lems, much less fixes, but at least we are not losing 
any more user e-mail, and our mail server is coping.) 


Before leaving ProcCountsChk, note that by 
defining all count thresholds to | across the board, we 
can guard against users running “‘dangerous”’ or “‘for- 
bidden” programs such as Crack or Sniffit. 


Case Study 5: UserActivity 
You can use PIKT’s define feature to achieve 


precision control over your security setup. Consider 
these security settings in defines.cfg: 


// lowest level 
jf Of Security 


attentive TRUE 


UserActivityLogChk 
Lace 
Status active 
evel cera ti ead 
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cautious 
#if misscritsys | cssys 
TRUE 
dFelse 
FALSE 
dFendif 
worried 
#if misscritsys 
TRUE 
dFelse 
FALSE 
dFendif 


paranoid FALSE // highest. level 


// of security 


Listing 10 shows how you might use them in an 
alarm to monitor suspicious, after-hours user activity 
(some per-OS customizations were omitted for 
brevity). 

We can also apply these defines to the UserAc- 
tivity.log file produced by the UserActivity alarm. 
Here is a sample log entry: 

Ago 3 <OVspoc Ol CRIM: root pO. -esh 


Listing 11 shows the log monitoring script. 


As security conditions change, we can generate 
more or fewer log entries by changing our security 


task "Report all new security incidents in UserActivity log." 


input logfile “=useractivity lop" 


#tifndef cautious // or worried or paranoid 


begin 
quit 
dtendifdef 


#tifdef cautious // or worried or paranoid 


rule 
output syslog Sinline 
dtendifdef 


#tifdef worried // or paranoid 
rule 
output mail Sinline 
ftelsedef // only cautious 


rule 
if JSimlane:—. “Toot” 
Output mail Sinline 
a: 
dtendifdef 
dtifdef paranoid 
rule 


OUTPUT print Sinline 
// page also? 
dtendifdef 


Listing 11: UserActivityLogChk. 
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defines (from TRUE to FALSE, or vice-versa) for dif- 
ferent systems, then using piktc to reinstall the modi- 
fiecl scripts on those systems. This gives us pinpoint 
control over our security setup. 


Other Uses 


Those are just a very few of the things you can 
use PIKT to monitor. We use it for all kinds of sys- 
tems administration tasks, including: clearing out /tmp 
files; reporting system crashes; monitoring changes in 
critical system files, directories, and devices; detecting 
passwd and shadow tile anomalies; running a mail 
quota system; reporting “orphaned’’ accounts and 
home directories; detecting bad e-mail list addresses; 
clearing out user Web browser caches; removing core 
files; rotating and retiring system log files; reporting 
full file systems; reporting runaway processes; reviv- 
ing vital system processes; reviewing security log files 
— the list goes on and on. 


Working with Other Scripting Languages 


If you prefer to use a different scripting lan- 
guage, that 1s no problem. Here is a short Pikt wrap- 
per script around a much longer, and very complicated 
Perl script, =mailchk (/pikt/lib/programs/mailchk. pl): 

MailChk 
aera 
Status active 
level warning 
task "Check Tor mai. \ 
errors; such as \ 
forwarding loops." 
input ‘proc =maiichk 2361" 


rule 
Output mail. Sinline 


mailchk.pl yields output which the Pikt MailChk 
script captures in a PIKT e-mail alert. If you wish, you 
could let your Perl script handle all the reporting, but 
still have PIKT deal with scheduling and logging, 
using the minimalist Pikt script: 


MailChk 


begin 


exec "=mailchk 2>&1" 


We have a suite of over two dozen account man- 
agement programs, almost all of them written in Perl, 
that we maintain within our programs.cfg file. We 
don’t use Pikt or piktd at all to run these. Rather, we 


#if usersys 
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use PIKT to manage the per-OS and per-machine dif- 
ferences, to install, and to monitor script integrity. 


Config Files Installation and Management 


Listing 12 is a portion of our files.cfg, the section 
configuring inetd.conf. 


Turning services on and off is as easy as editing 
the central files.cfg, then reinstalling inetd.conf with 
the appropriate piktc command. 


Recently, a CERT advisory was broadcast advis- 
ing against running the rpc.ttdbserverd service with 
root privileges. For the rpc.ttdbserverd line, we substt- 
tuted “daemon”’ for “root”’ (the line was already com- 
mented out anyway), then updated inetd.conf and 
reconfigured inetd on all Solaris systems with: 


if pikte ivr anetd.cont \ 
+H solaris -H downsys 


## piktc -xv +S SigHupInetd \ 
+H solaris -H downsys 
where SigHupInetd 1s a Pikt script written expressly 
for that purpose. 


Another problem we have faced is keeping up- 
to-date our sudoers file — especially the list of part- 
time Computer Assistants. We do it in files.cfg by 
means of an include file: 


User_Alias PARTTIMERS=\ 
#include <sudo_parttimers_files.cfg> 


where the sudo_parttimers_files.cfg file might be: 
larry<moe,curly.sporty,\ 


ginger, baby ,posh,scary,\ 
john, paul, george,ringo 


We have a separate script that rewrites the 
sudo_parttimers_files.cfg file mightly based on an 
authoritative and up-to-date GNU Mailman list. The 
result: a dynamic sudoers config file! 


Remote Command Execution 


You can use piktc for remote program execution 
as an alternative to rsh or ssh. The command 
dt pikte.-Xv-tG "<command(s) >> 
+H <systems>? 
executes the given command(s) on the specified sys- 
tems. | 


You can insert PIKT macros within +C com- 
mand strings. See Listing 13, for example. 


FOO fUSsritepd;scpa. anertpd 
root fusr/tcpd/tcepd in.telnetd 
root fusrecepd(tepd <inattpd 
root jusritepd/tepd in. telnetd 


Listing 12: files.cfg (fragment). 


Ee Stream tcp nowait 
telnet stream tcp nowait 
Helse 
#£tp stream tcp nowait 
telnet stream tcp nowait 
fHendif 
160 
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Note that kiev0 is a SunOS system. We want ‘df 
-k’ to run on the Solaris systems and just plain ‘df’ to 
run on the SunOS systems. The macro =dfk resolves 
to the desired path and command option. 


System Lists 


With perhaps the simplest but still useful PIKT 
setup imaginable — the piktc binary and a systems.cfg 
file — you can maintain custom system lists, whether 
for referencing within other programs, as in this Perl 
statement 

@hpsys = ‘piktc -L +H hpux \ 
=H downsys 
or for command-line work, as in a command loop we 
use to upgrade our Solaris PIKT binaries (see Listing 
14). 

The uses of PIKT really are limited only by your 

imagination! 


Security 


The current PIKT security model is fairly trust- 
ing. There are things you can do now to tighten secu- 
rity, while other things — Kerberos-style client-server 
authentication and data encryption, for example — are 
under study and planned for implementation in the 
near future. 


In PIKT.conf, which functions roughly like a 
.thosts or .shosts file, you set various access parame- 
ters (e.g., uid, gid, master, domain, master_address, 
etc.) and service rights (e.g., all services, kill service, 
install service, etc.). By careful and judicious use of 
these settings, and in combination with other measures 
(firewalling, running piktc_svc only when necessary, 
etc.), you can achieve a level of security sufficient in 
many situations. 


When you issue a piktc command, the slave 
(remote) host and requested service are registered with 
the master (local) piktc_ svc. Upon receiving the 


## piktc -x +C "hostname; =dfk /tmp" +H mus 
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service request, the slave piktc_ svc checks its local 
access authorizations in PIKT-.conf. If the service 
request is authorized, the slave piktc_svc then does a 
“callback” — a second, independent TCP connection 

— to the master piktc svc, seeking to verify the 
request. If it verifies — 1.e., both slave host and service 
request match — only then does the slave piktc_svc 
perform the requested service. It then returns the out- 
come of the service request to the requesting piktc. 
Finally, the slave host and requested service are dereg- 
istered on the master piktc_svc. Please see Figure 1. 


piktc and piktc_svc do complete service request 
logging, and piktc_svc marks denied requests as 
““ERROR”’. It is not difficult to set up log monitoring 
scripts (and scripts to monitor the monitoring scripts, 
etc.) to spot attempted security break-ins. Remember 
that you can log to syslog and special logs, and dump 
to a printer, besides sending out e-mail alerts. Scripts 
can also call pagers in security emergencies. You can 
even have a monitoring script kill the local piktc_svc 
under suspicious circumstances. 


You can use PIKT for security in at least the fol- 
lowing six ways: 

1. As just a centrally managed scheduler. Have 
piktd invoke your preferred security tools 
according to the schedules in piktd.conf. 

2. The above, plus have PIKT manage other secu- 
rity tools’ config files (the inevitable per- 
machine and per-OS customizations). 

3. All of the above, plus use PIKT #ifdef’s to acti- 
vate and deactivate different security tools as 
changing conditions warrant. 

4. All of the above, plus use PIKT to handle your 
security log file analysis and incident reporting. 

5. All of the above, plus employ PIKT alarms and 
data objects as supplements to the standard 
tools. (For example, have PIKT do things that 
COPS or Tiger don’t do.) 

6. Use PIKT exclusively. 


kievO 

Filesystem kbytes used avail capacity Mounted on 

/dev/sd0e 993006 17, 893669 O% /tmp 

kiev 

Filesystem kbytes used avail capacity Mounted on 

swap 769096 296 768800 1% /tmp 
Listing 13: piktc Command Example 

# for sys in ‘piktc -L +H solaris -H piktdevsys no_usr_local downsys’‘ 

> 0 

> echo Ssys 

> Ssh osys’ '/pikt/ ii b/ programs/svestart. pl “=k: \ 


ep ipakt/ban/pikt” /pikr/ band? bake. \ 


ep. /usrPlocal)/ piakt/bin/solaris/pakr*: /pikty bing -\ 


fpaktiiiby programs ;svestar tpl. ort 
> done 


Listing 14: System Lists 
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PIKT is especially adept at points one, two, and 
three above. 


As for point four, there are good solutions out 
there already for analyzing your security log files, but 
PIKT might be superior due to its more powerful and 
flexible built-in scripting language. 

As for point five, extending one tool requires you 
to lear Perl, another to get intimate with the Bourne 
shell, while five others require you to learn five differ- 
ent cryptic and proprietary command languages. Learn 
those languages and modify those tools on their own 
terms where that makes the most sense, but when sen- 
sible resort to PIKT. 


As for using PIKT exclusively, in spite of its 
great virtue of involving just one system and com- 
mand language to iearn and use, | don’t propose that 
PIKT do everything! For many purposes, there are 
some really excellent alternatives available. | therefore 
suggest that the optimum solution lies somewhere 
around points three, four or five. 


Security should be systematic, flexible, and easy 
to manage. These are areas where PIKT excels. At the 
very least, PIKT is a general framework on which to 
build your security efforts. 


Future Plans 


PIKT could use a GUI, not so much to overlay 
pikte management as to handle incoming alert mes- 
sages. This could take form as, for example, a Tk/Tcl 
or Java front-end acting on syslog messages sent to the 
console machine. 


Although there are methods to program against 
endlessly repeating nuisance messages (so-called 
‘“‘nagmail’’), PIKT would benefit from more auto- 
mated ways to do this. Message routing could also be 
improved. 


PIKT has a steep learning curve, and setup can 
be daunting. A project is underway to put together a 
PIKT “‘standard library” of ready-to-run defines, 
macros, alerts, scripts, programs, and objects sets. We 
need to improve the user’s “out-of-the-box” experi- 
ence. 


The weakest link in PIKT’s chain 1s the script 
interpreter, pikt. Although it gets the job done, it is not 
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fast or feature-complete. Pikt is not a stand-alone, all- 
purpose scripting language, and you cannot etfectively 
call Pikt scripts from programs written in other script- 
ing languages. Pikt was designed to aid in systems 
administration and to run “fast enough” in a small 
memory space within the broader PIKT system. Plans 
are to rewrite Pikt’s underlying engine, perhaps using 
GNU Guile or embedded Perl. 


PIKT requires a thorough security audit, and its 
client-server communications need to be made iron- 
clad secure. Adding encryption and Kerberos authenti- 
cation is under consideration. We also plan to imple- 
ment a comprehensive security package of PIKT 
defines, macros, scripts, and objects sets to work alone 
or in concert with other security products. 


PIKT has an Introduction and comprehensive 
Reference Manual but lacks a Getting Started guide, 
also an Operations Manual. 


Other Solutions 


PIKT is often compared to Cfengine [3]. In the 

words of its author, Mark Burgess, 
Cfengine ... is a very high level language for 
building expert systems which administrate and 
configure large computer networks. Cfengine 
uses the idea of classes and a primitive form of 
intelligence to define and automate the configu- 
ration of large systems in the most economical 
way possible. 


In Cfengine, you create a configuration file (or 
files) describing the ideal setup for all of your hosts. 
When run, the cfengine program will check the actual 
machine configurations against the ideal and, if 
desired, fix any deviations. 


Cfengine and PIKT address generally the same 
problem but in significantly different ways. Cfengine 
is a high-level, declarative or descriptive language (a 
single statement might set permissions on hundreds of 
files, for example), while Pikt is a low-level, procedu- 
ral language. Cfengine tends to provide specific solu- 
tions to specific problems, while PIKT tends to be 
more general. Cfengine’s specificity (it has built-in 
support for configuring network interfaces, for exam- 
ple) would be out of place in base PIKT. (With PIKT, 


SLAVE 


| (2) send request 


| 
| 
| 
| 
V 


piktc_svc<------------- 
(4) callback 


(5) reply 


(7) seturn result 


(1) register slave & request | 
(8) deregister slave & request | 
V 


pier eit ee Re Ppikteessvc 


(3) “check. authorizations 
(6) perform service 


Figure 1: callback. 
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you would write a script to configure the network 
interface calling the usual UNIX networking com- 
mands.) 


While Cfengine achieves per-OS and _ per- 
machine customization by means of classes, for exam- 
ple, 

FTPserver.Sunday.Hr0O0:: 
/local/iu/xferlog rotate=3 


which means to run xferlog at midnight Sunday if this 
system is a FT Pserver, PIKT would achieve a similar 
effect as follows (in the alerts.cfg file): 


Notice 
timing De ee 20 
alarms 
#if ftpserver 
LogFileChkNotice 


fFendif 


Cfengine has its_own unique keywords, syntax, 
macro and variable forms, etc. Although Pikt has 
some unique elements, much of it should be familiar 
to any Perl or C programmer, especially the idea of 
preprocessing. If PIKT has a steep learning curve, 
Cfengine’s is equally steep, if not steeper. 


Cfengine tries to anticipate many of your needs, 
but when you veer off the beaten path, Cfengine is not 
quite so helpful. In many situations, you will still need 
to write your own scripts. With PIKT, you script 
everything. This makes PIKT inherently more flexible 
and applicable to a broader class of applications — not 
just fixing broken system configurations and execut- 
ing routine tasks, but also reacting to errant dynamic 
processes. 


Cfengine is quite good at what it is designed to 
do. It would be especially useful (and superior to 
PIKT) for configuring a new system or restoring a sys- 
tem after a crash or cracker break-in. One really nice 
Cfengine feature is that ordinary users can invoke it, 
attempting to fix a broken configuration if the system 
administrator is unavailable. (PIKT is typically just for 
root use.) 


In work first presented at the LISA 1999 Confer- 
ence [4], Alva Couch and Michael Gilfix have 

.. created a system administration library that 
allows one to perform system administration 
tasks in Prolog. This is much more powerful and 
flexible than using other current tools, and has 
the advantage that the resulting Prolog programs 
are much closer to describing actual policies than 
CFEngine configuration files or PIKT scripts. 


Perhaps because their comments were based on 
earlier, less mature versions of PIKT, | feel that they 
underestimate the power, flexibility, and expressive- 
ness of Pikt scripts, especially the fully documented, 
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macro-enhanced versions found in the central configu- 
ration files (as opposed to the preprocessed, uncom- 
mented versions installed on the slave systems). 


Their Prolog-based approach to systems adminis- 
tration is intriguing and potentially far-reaching, but it 
suffers from one significant problem: Unless one 
attains proficiency with Prolog (not a widely used lan- 
guage, to say the least), their system is a “black box,” 
closed to the do-it-yourselfer who demands complete 
control over, or at least complete understanding of, the 
system. In any case, at this time, source code is not yet 
available for public distribution, so it is hard to evalu- 
ate their approach effectively. 


There are other systems monitoring packages out 
there, including: Big Brother [7], and its clone Big 
Sister [1]; Mon [9]; NetSaint [5]; and still others. 
These tend to focus more on performance statistics 
and problem reporting, less on systems configuration 
and problem solving. To their credit, they rely on 
standard scripting languages, but they don’t deal 
specifically or as extensively with the problem of per- 
machine and per-OS customization like PIKT, 
Cfengine, and the Prolog-based library do. 


I have no experience using any of the high- 
octane, very expensive commercial packages (like 
Tivoli [8] or CA Unicenter TNG [10]) and can’t ven- 
ture any comparisons or opinions about them. 


Parting Thoughts 


The heart and soul of PIKT is its preprocessor, 
piktc, and all the special scripting and file manage- 
ment facilities it provides: per-machine and per-OS #if 
directives; the #ifdef family of logical switches; 
#include files; macros; pinpointed file installation; 
central scheduling; and so on. PIKT moves scripting 
toward the kind of full-featured development environ- 
ment that users of “‘more serious” languages have 
long enjoyed. 

The Pikt scripting language offers some unique 
features, or features better tailored to the job of day- 
to-day systems administration: automatic previous- 
line (@foo) and prior-run (Y%foo) value references; a 
clean, uncluttered syntax; free-form, flexible layout; 
keyword synonyms; automatic logging of everything 
of consequence; a cautious approach to script execu- 
tion (no assumed variable defaults; serious errors trig- 
ger automatic script shutdown); a built-in input loop 
(much like AWK’s); many standard input and output 
options. 

On the other hand, people have a right to ques- 
tion whether the world needs Yet Another Scripting 
Language. Also, scripting language preference is often 
a highly personal, even emotional matter. Pikt, the 
scripting language, is just the first among equals in 
PIKT, the sysadmin toolkit. Use of other languages 
within the PIKT system 1s encouraged, and embedding 
other scripting languages within PIKT is actively 
being considered. 
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But the point bears repeating: the piktc prepro- 
cessor and control program 1s at the center of PIKT. It 
is what sets PIKT apart from other scripling languages 
and other system monitors. PIKT is more than just 
another systems monitor and Yet Another Scripting 
Language. When confronted with its multi-functional- 
ity, one Web administrator didn’t quite know where to 
list #t, saying that he might have to invent a whole new 
category for PIKT. If ever a tool were more than the 
sum of its parts, PIKT is that tool. The PIKT combina- 
tion is a very powerful, wide-ranging, and ambitious 
toolkit indeed. 


Availability 


PIKT’s homepage 1s: http://pikt.uchicago.edu/pikt, 
where you will find not only the distribution package 
but also complete on-line documentation, sample con- 
figuration files, a comprehensive test suite (with over 
600 validation tests), and other useful items. PIKT is 
also available for download at a number of ftp sites. 
pikt-users and pikt-workers mailing lists have been 
formed. Operating systems now supported include 
GNU/Linux, Solaris, SunOS, FreeBSD, OpenBSD, 
AIX, HP-UX, and IRIX. 
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Appendix 1: piktc Command Options 
Usage: piktc <-cCdefGhikKlLm#rRstTvxX> 


[Fl <p <define(s)>] 

[+C <command(s) >] 
[+|-A|S[f]) all]<alert(s)/script(s)>] 
fry 2 (£] all}<program(s)>] 
Eey-E(f] allfi<file(s)>] 

[*] -O[f) all|<object(s)>] 

+|-H all }Shost (s)> 

PAT 

iC syntax check all config files 

=C syntax doublecheck all config files 

aid disable alert(s) 

-e enable alert(s) 

-f diff alert/script/program/file/ 
object file(s) 

aS run in debug mode 

=1 show program help 

-4 instal l-alert/seript/ program? file/ 
object file(s) 

-k kill alert daemon (piktd) 

=k kill service daemon (piktc_svc) 

a list alert/seript/program/ files 
object file(s) 

-L list alert/script/program/file/ 
object or host/os/group/alias 
command-line item(s) 

- mit checksum alert/script/program/file/ 
object file(s), where # is 
checksum level 1-5 

= (re)start alert daemon (piktd) 

-R (re)start service daemon (piktc_svc) 

-s show alert(s) status 

= delete alert/script/propram/file/ 
object/history/log file(s) 

oT run in test mode 

ay run in verbose mode 

xX execute alert(s)/script(s) 

ax execute alert(s)/script(s) 
with no wait 

Flee <define(s)> define/undefine define(s) 

ie <command(s)> include command string(s) 

aL [f] all|<alert(s)> inelude/exclude (fix) alert(s) 
=P ral all|<program(s)> include/exclude (fix) program 
file(s) 

+ obitd all|<file(s)> include/exclude (fix) other 
file(s) 

Hy Od all|<object(s)> include/exclude (fix) object 
file(s) 

+|-H all|<host(s)> include/exclude host(s)/os(es)/ 
group(s)/alias(es) 

ALL +A all +S all +P all +F all 


+O. ae sae: 
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Relieving the Burden of System 
Administration through Support 
Automation 


Allan Miller & Alex Donnini — HandsFree Networks 


ABSTRACT 


The number of computer users with little or no training continues to rapidly increase. 
Networks are at the heart of companies large and small. Applications, ever more complex, span 
intranets and extranets. This all adds up to an increasing burden on system administrators and end 
user support organizations at a time when there is constant downward pressure on support budgets 
and a shortage of qualified staff. The result is a technical support crisis with dissatisfied end users, 
bumed out system administrators, and unhappy support teams. The need for support automation is 
critical as it enables the scaling of effective end user support while minimizing the need for 
additional resources. 


There are significant challenges to automating support, since it 1s essentially a large 
collection of special cases. Different methods have been used to achieve this automation, with 
varying degrees of success. This paper describes a software system that automates the solution of 
many recurring end user problems, greatly relieving the burden on system administration staff for 
mundane issues. We describe the architecture of the system, give examples of its use, demonstrate 


its extensibility, and report on our experience using it in the field. 


Introduction 


The widespread use of computers in mission- 
critical functions continues to grow, resulting in their 
use by ever-larger numbers of technically unsophisti- 
cated end users. At the same time, the software 
installed on them gets more complex as competitive 
pressures force vendors to incorporate additional 
advanced features. To make matters worse, the infras- 
tructure itself is becoming inherently more sophisti- 
cated: a seemingly simple application such as email 
involves the use of network connectivity and routing 
that is far beyond the capabilities of an end user to 
diagnose and repair. Yet budgets typically do not leave 
room for additional supp ort needs, as demonstrated in 
[Tol92]. The increasing demand for support services 
by end users puts system administrators in a position 
where they have to solve more and more repetitive 
Issues, In addition to dealing with the more complex 
enterprise-wide issues they face every day. Often, they 
are simply not in a position where they can handle this 
additional burden. 


Outside the enterprise, the situation 1s even 
worse. With less access to system administration 
resources, smaller businesses rely on a combination of 
internal resources and third party service providers. 
The labor intensive support process used by third 
party service providers simply does not scale to the 
level being demanded by current and future use. As 
evidenced in [DeK99], smaller businesses that cannot 
afford a full-time system administrator are increas- 
ingly dissatisfied with their support. As software 
becomes embedded in more and more consumer 
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products (such as cell phones or even refrigerators), 
the problem continues to worsen and is beginning to 
affect home users as well as businesses. 


As a result, there 1s an urgent need for an 
increased level of automation in the process of every- 
day administration, maintenance, and support tasks for 
end users. System administrators must be freed from 
the overwhelming load of essentially trivial problems 
that recur on a frequent enough basis to interfere with 
the solution of more important and challenging prob- 
lems. This paper describes a software system that 
automates the solution of many recurring end user 
problems, greatly relieving the burden on system 
administration staff for mundane issues. 


Existing Support Automation 


One of the indications of the need for automation 
of end user support is the existence of a range of prod- 
ucts in this space. These products all address the need 
to some point, but have various shortcomings that are 
addressed by the system described in this paper. 


There are a number of products that detect sys- 
tem level events, such as processor faults, and attempt 
to recover from their effects. These products use a 
general mechanism and apply it to all instances of the 
problems being detected. However, the general mech- 
anism really only addresses the immediate cause of 
the problem event, and does nothing to permanently 
address the root cause of the underlying problem. 


Some products provide automated software 
updates to customers. This solves problems caused by 
bugs that are fixed in a later release of the software. 
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However, many end user issues are not caused by 
bugs, but are instead usability issues where the soft- 
ware 1s working as designed but the designers have 
not foreseen its obscure behavior in a common cir- 
cumstance. In addition, most problems can be 
addressed with workarounds that end users can apply 
immediately, eliminating the need to wait for the next 
turn of the software development cycle of the product. 


Virus scanners are important tools for every sys- 
tem administrator. They detect system level events and 
changes to files, use a database to drive the detection 
and removal of viruses, provide reporting functions on 
virus detection and elimination, and periodically 
update the database automatically. As we will see, the 
system described in this paper is similar in some 
respects to a virus scanner. However, virus scanners 
have a very specific problem domain on which they 
focus. They provide no help to a system administrator 
for the myriad of other end user issues that arise on a 
daily basis. 


“*Self-healing” systems provide a facility, either 
manually guided or semi-automated, to restore some 
or all files back to a previous state. This can be a 
powerful device for quickly getting a system back to a 
working condition. However, it is most helpful in a 
disaster recovery situation, and is not of much help 
when a problem must be solved rather than removed, 
or when a probiem has existed for a long time without 
manifesting itself. 


Some “web-enabled support” sites provide a 
database of known problems and solutions (usually 
called a “knowledge base’’). [Mic00] is a good exam- 
ple of a well-built knowledge base. While knowledge 
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bases have demonstrated their value in providing 
quick access to vital information for system adminis- 
trators, the concept of their direct use by end users 
leaves much to be desired. Most end users have diffi- 
culty finding and understanding information in a 
knowledge base, and may actually end up doing more 
harm than good by attempting to apply inappropriate 
solutions. In addition, moving the task of system man- 
agement into the hands of an inexperienced end user is 
not a very popular option in an enterprise environment 
where “‘time 1s money.” 


Help..desk software is a key component of an 
enterprise solution for end user support. Effective help 
desk software can increase the efficiency of an internal 
support group and therefore reduce some of the bur- 
den on system administrators. In addition, some help 
desk software can assist an internal support group in 
building a database of problems and solutions, which 
can be extremely useful to administrators. However, 
the software is of little direct use to those system 
administrators by itself. 


Some help desk add-ons allow end users to con- 
tact a support group electronically, using email or 
chat, and facilitate problem resolution through the use 
of advanced diagnostics. Some of these products also 
feature limited automation of problem resolution 
activities and remote support capability, allowing a 
support representative to take control of an end user’s 
desktop to resolve a problem. This removes some of 
the potential for human error in diagnosing and resolv- 
ing the problem. These tools can certainly help a sys- 
tem administrator maintain a larger number of end 
users more efficiently. However, even with these tools, 
each end user issue requires individual attention, so 
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the system administrator is still forced into the ineffi- 
cient role of providing repetitive end user support on 
common problems. 


“Expert marketplaces”’ match up end users with 
consultants who are bidding services to solve desktop 
computer problems. This is an interesting “free mar- 
ket” approach to providing end user support by out- 
sourcing it to a widely distributed support staff. How- 
ever, it does suffer from somewhat inconsistent 
results, due to the variety of talent bidding the support 
services. It is easy to imagine a situation where multi- 
ple consultants would inadvertently provide conflict- 
ing point solutions to a system wide problem, merely 
interfering with the efforts of the system administra tor 
who is ultimately responsible for the overall solution. 


Finally, a number of “support portals” are avail- 
able that feature online access to some combination of 
the services listed above. These provide convenient 
“one stop shopping”’ for those services, but do not 
otherwise enhance the automation or scalability of the 
services themselves, and are typically geared to a 
lower level of expertise than that of a system adminis- 
trator. 


In summary, while these approaches are all valid 
and help with some aspect of the end user support 
issues faced by system administrators, none of them 
really shorten the administrator’s lengthy task list by 
automatically detecting and resolving simple recurring 
problems without any manual intervention. The sys- 
tem described in this paper addresses many of these 
shortcomings and provides a true solution for saving 
time and lightening the load on the system administra- 
tor. 


Automating Support 


Figure 1 shows the traditional process for han- 
dling end user problems, as exemplified in [RSAO0O] 
and [Sm197]. The process starts when an end user 
determines that a problem is happening. The user 
contacts! the system administrator. Through a discus- 
sion with the user, the administrator clarifies the 
nature of the problem, then searches through a 
database of pre-defined problem descriptions for a 
match. In many cases, the “‘database” is simply a 
mental list of common problems and solutions. It can 
also be an informal written list, or if the system 
administrator is using help desk software to manage 
the process, it may be an actual database. If a match is 
found, the administrator communicates the resolution 
to the user, and assists the user in applying it. [Etc98] 
describes that this process works well, solving as 
many as 80% of the problems encountered by end 
users [Sla00, Gil00, Sup00, SchO0, Hon00, CWS99, 
Loc00]. For the remaining 20% or so, the system 
administrator must become directly involved in the 


'This could be a personal contact or a telephone call. In- 
creasingly, other forms of interactive electronic communica- 
tions are being used. 
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diagnosis and solution of the problem, and may need 
to escalate the process to include the software vendor. 
If the problem is escalated, the vendor uses product- 
specific knowledge to reproduce the problem and 
attempt to resolve it. If the resolution is successful, the 
procedure for resolving the problem is communicated 
back to the system administrator, who in turn commu- 
nicates the resolution to the end user and assists in its 
application. In addition, the system administrator now 
adds this new problem and resolution to whatever 
“database” is used for solving common problems. In 
that way, new instances of the problem can be 
resolved more quickly, without needing either direct 
involvement or the assistance of the vendor. The 5% 
or so of problems that cannot be resolved by the ven- 
dor’s customer support are escalated internally to the 
engineering team at the vendor, where development 
resources are brought to bear on diagnosing and 
resolving the problem. In this way, the efforts of the 
system administrator and the vendor’s customer sup- 
port group play two critical roles in the process: keep- 
ing the “database” populated with relevant problems 
and solutions for common recurring problems, and 
providing qualified, valid bug reports to the vendor’s 
engineering staff. 


Clearly, the amount of human effort involved in 
Figure 1 makes it a very labor-intensive and error- 
prone process. To make things worse, the ‘“‘database”’ 
in Figure 1 may be simply mental notes, so the system 
administrator may be the only person who can drive 
problem resolution. Even with a more formalized pro- 
cess, the quality of the response is very dependent on 
the ability of the person using the database, whatever 
its implementation. 


Figure 2 shows the process for automated sup- 
port. A software client manages the process for the 
end user. When the client detects that a problem is 
happening, as described later, it searches a local 
database for a match to the symptoms of the problem. 
If a match is found, the database contains executable 
code that the client runs in order to resolve the prob- 
lem. Since the database is functionally equivalent to 
the database in Figure 1, this solves as many as 80% 
of the problems encountered by the user. For the 
remaining 20% or so, the client forwards a detailed 
description of the circumstances surrounding the prob- 
lem to the system administrator, who 1s made aware of 
the problem early on. At that point, the administrator 
may choose to solve the problem or may instead want 
to escalate it to the vendor. In either case, the informa- 
tion from the client can be used to diagnose, repro- 
duce, and solve the problem. When a solution is 
found, either the system administrator or the vendor 
codes and tests a solution for the problem and adds it 
to the master database. This master database is used to 
update the local database, which the client then uses to 
resolve the problem. The 5% or so of problems that 
cannot be resolved by the system administrator or ven- 
dor’s customer support group are escalated to the 
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vendor’s engineering staff. The system administrator 
and vendor’s support group continue to play the same 
two critical roles as in Figure 1: keeping the master 
database populated with relevant problems and solu- 
tions for common recurring problems, and providing 
qualified, valid bug reports to the vendor’s engineer- 
ing staff. 


It is worth pointing out that some problems can- 
not be detected, and some solutions cannot be imple- 
mented, by the system outlined in Figure 2. The client 
cannot automatically detect problems that completely 
disable the system, such as a faulty power supply, 
since the system cannot run the client. The client can- 
not automatically implement a solution that requires 
physical modification of the system, such as replacing 
memory. Likewise, the client cannot automatically 
detect problems that are not amenable to software 
detection, such as poor quality in printer output. 
Finally, the client cannot automatically resolve prob- 
lems that are caused by a misunderstanding on an end 
user’s part, such as inability to set the margins in a 
word processor. However, two powerful techniques 
can be used to overcome some of these limitations. 
The first is the use of communications between clients 
on multiple machines. Even if a problem cannot be 
detected directly on the machine where it occurs, its 
external effects may be readily identified on other 
machines in the network. The second is the use of an 
interactive dialog with the user of the machine. Even 
when the client cannot test or modify certain aspects 
of its environment, it may be able to instruct the user 
to do so on its behalf. In the end, even if the client 
cannot completely diagnose or resolve a problem, the 
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information it gathers in doing so will be immensely 
helpful to the system administrator ultimately respon- 
sible for the problem resolution. Our analysis, 
described later, of the most frequently occurring prob- 
lems in the Microsoft Knowledge Base, [Mic00], indi- 
cate that software configuration issues cause the vast 
majority of these problems. This means that the kinds 
of problems the client cannot address make up a rela- 
tively unimportant fraction of those that actually hap- 
pen. Anecdotal evidence also seems to support this 
conclusion. 


Two important features of automated support are 
evident in Figure 2. The first is that up to 80% of the 
problems that occur on the end user system are 
resolved quickly and accurately, without any human 
intervention. The user may not even be aware that the 
problem existed in the first place. The second is that 
the tasks of the system administrator are driven by the 
appearance of new problems, not by the appearance of 
new users. This means that the part of Figure 2 that 
needs to scale up the most to handle more end users is 
the update of the local database from the master 
database. An increase in the number of users does not 
require a proportional! increase in the system adminis- 
tration group, so large numbers of users can be sup- 
ported without a huge staff. 


There are significant challenges to automating 
support. By definition, the task is a large collection of 
special cases, but software tends to work best with 
small sets of algorithms applied to large numbers of 
uniform tasks. In addition, the applications that are 
being supported cannot implement the support them- 
selves, since they are presumably malfunctioning. 
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Since nearly any aspect of system operation can be 
part of the problem, and part of the solution, the auto- 
mated support tool must be as independent as possible 
from the system and its applications. At the same 
time, 1t must be aware of most of the details of the sys- 
tem operation. 


These factors make the job of automating sup- 
port difficult, but not impossible. Database techniques 
can be used to manage the large number of special 
cases. Both the diagnosis of problems (“‘what happens 
when you run...?”) and their resolution (“now 
change the setting...’’) are essentially software tasks 
that can be automated. Combining these two consider- 
ations, the implementation of automated support 
works well with a database containing executable code 
for the diagnosis and resolution of problems. The size 
of the database is strongly affected by the ‘80-20 
rule”: 80% of end user issues that arise can be solved 
using only 20% of the entire universe of solutions. 
(Note that this is a little different from the 80% and 
20% mentioned above, which referred to the fact that 
80% of user issues can be resolved using a database.) 
The 80-20 rule is well known in support circles 
[Che99, Che99a, Gia99, Lan00, Ste99, Sum98]. The 
implication is that a relatively small database can be 
extremely effective in resolving user issues. We now 
describe the use of such a database in an automated 
support system provided by HandsFree Networks. 


System Architecture 


The heart of the system is the client that runs on 
each end user machine in the facility. Figure 3 shows 
an overview of the client. The database contains a col- 
lection of scrips,? each of which contains executable 
code to diagnose and resolve a single problem. A scrip 
consists of four major parts: initialization, configura- 
tion, symptom, and solution. These major parts are 
described in more detail later in this paper. 


2The term “scrip” is borrowed from the medical profes- 
sion, Where it is the working jargon for a prescription. Like 
these database entries, pharmaceuticals are used both for di- 
agnosing and resolving problems. There is also a connec- 
tion to the word ‘‘script.” 
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Event detection drives the application of scrips to 
problems based on system level events such as win- 
dow creation, user input, processor faults, process cre- 
ation and termination, device notifications, and timer 
expiration. The event detection module efficiently 
determines which, if any, scrips should be run as a 
result of a sequence of system level events. It is 
designed to very quickly dismiss events that do not 
trigger a scrip. Since the resulting conceptual model of 
a scrip is an interrupt service routine rather than a 
polled service, a scrip only uses system resources 
when a problem is detected, thereby minimizing its 
impact on the system. 


The execution scheduler is responsible for 
dynamically loading and executing the code for a scrip 
on an as-needed basis. It is in charge of managing the 
synchronization of scrips that are running in multiple 
threads, as well as passing information from detected 
events to the scrips. The database also contains proce- 
dures that are dynamically loaded when they are 
called from a scrip; the execution scheduler manages 
the loading and execution of these procedures, as well 
as the passing of parameters into and out of the proce- 
dures. 


Primitives are a library of pre-defined routines 
that provide a convenient (and, where possible, system 
independent) interface to operating system functions. 
They also provide an interface to common utilities 
such as memory allocation and string manipulation. 
Scrips and procedures in the database use primitives to 
accomplish most of their work. From the perspective 
of a scrip, the interface to a procedure is the same as 
the interface to a primitive, so procedures in the 
database serve as a convenient and flexible way to 
extend the primitives. 


Scrips use the persistent state mechanism to keep 
information that must persist across multiple invoca- 
tions of the scrip and machine restarts, as well as 
information that must be accessed by more than one 
scrip. Variables in persistent state have a namespace 
that limits their scope, so a scrip author can freely 
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define and use a persistent state variable without con- 
cern for interfering with the variables in other scrips. 


The remote interface allows scrips to run code 
with equal ease on any machine in the network, 
including the machine on which the scrip is currently 
running. This is accomplished by making every call to 
a primitive specify the machine where it should run. 
An example of this is described in more detail later in 
this paper. 

A key feature of the client is its portability to 
multiple environments. It has been designed from its 
inception to have well-defined system dependent and 
system independent modules with consistent inter- 
faces. One of the authors has architected three major 
portable software systems in the past (described in 
[Vir97, Ter00, CAS87]), and has brought relevant 
experience to bear on this system as well. The current 
version of the client runs on many implementations of 
the Win32 API (Windows 95, Windows 98, Windows 
NT 4, Windows 2000, and Windows Me), and a Linux 
version is presently in development. 


Much of the value of the system comes from the 
fact that the scrip database is updated on a regular 
basis, since end user issues change regularly as new 
products are introduced and new problems are discov- 
ered. As a matter of fact, one of the greatest benefits 
of implementing the client using a database architec- 
ture is the flexibility in providing this update. For 
example, the scrip database can be implemented as a 
central database, or a central database with local 
caching, or a distributed database, or a redundant 
database, or any combination of those options. In this 
application, a problem that affects network connectiv- 
ity may isolate the client from the network, so a stan- 
dalone local database is desirable. A simplified repre- 
sentation of the default product configuration is shown 
in Figure 4. A centralized master scrip database 
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contains all available scrips. At the system administra- 
tor’s facility, a local scrip database stores the subset of 
the master scrip database that is relevant to the local 
hardware and software configuration. At a regular 
interval (about once a week), the client on one of the 
administration machines initiates an incremental 
update of the local database to retrieve any new or 
modified scrips. All the machines at the facility 
access the local scrip database for the latest versions 
of all scrips. In addition, they also have a small 
database of scrips to solve connectivity that is repli- 
cated on every machine. If a machine cannot access 
the local scrip database due to a connectivity issue, 
this database helps to solve that problem first. Note 
that both the master scrip database and local scrip 
database can be replicated for improvements in relia- 
bility and performance. 


The implementation of security in the system is 
of paramount importance. Since the primary function 
of the software is to transmit and run executable code, 
it would be an ideal virus infection vector. To prevent 
this sort of abuse, all network access goes through the 
remote interface and uses the HTTP protocol. The 
remote interface implements SSL as described in 
[Fre96]. For maximum security, it can be configured 
to require both client and server authentication. This 
encryption protects system integrity by preventing 
malicious attacks on the code executed by a client, and 
also protects data security by preventing passive 
observation of the data communicated between two 
clients while executing a primitive remotely. In addi- 
tion, all scrips are digitally signed, and the engine can 
be configured to only run scrips from a list of trusted 
providers. Even without strong (or any) encryption, 
the digital signatures prevent the execution of malli- 
cious code by a client, since the attacker can forge a 
scrip but cannot forge its signature. This protection 
works for worldwide deployment of the database, 
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since export restrictions on strong security only apply 
to data encryption, not to digital signatures. 


Interestingly enough, the connotation of a “‘cer- 
tificate authority” for the SSL security in the client is 
somewhat different from the normal one. In secure e- 
commerce transactions, a certificate authority 1s sin+ 
ply declaring that the identity of the entity being certi- 
fied is actually correct. A certificate authority for 
scrips, on the other hand, is declaring that the entity 
being certified is one that can be trusted to produce 
reasonably stable and desirable scrips. Presumably, the 
certificate authority provides this service by verifying 
that the entity has reasonable QA procedures in place, 
doing periodic audits of the software production facil- 
ity, and doing random independent code reviews and 
testing of the scrips themselves. This is quite a bit 
more complicated than simply verifying identity, but 
is worth correspondingly more as a value added ser- 
vice. HandsFree Networks plans to provide this ser- 
vice as part of its product offering, but also expects 
that third party providers will provide a similar ser- 
VICE. 


Problem Resolution Process 


Referring to Figure 2, the problem resolution 
process starts when the event detection mechanism 
recognizes a certain sequence of system level events 
that corresponds to an entry in the scrip database. It 
puts together relevant information about the system 
level events into an “‘event” that 1s passed to the exe- 
cution scheduler, along with an identifier indicating 
which scrip is being triggered. 


The execution scheduler decides when the trig- 
gered scrip should be run and loads the scrip symptom 
from the database. It runs the code for the symptom, 
which does any kind of verification needed beyond the 
trigger to ensure that the problem is indeed present. If 
the symptom indicates that the problem ts present, the 
execution scheduler loads the scrip solution from the 
database and runs the code for it. 


When a problem cannot be resolved locally, an 
escalation process prevents the situation where an end 
user is left without a solution. Escalation can be initi- 
ated by the failure of a solution, which is detected 
when a scrip continues to diagnose a problem even 
after its resolution has been applied. Escalation can 
also be initiated by a “diagnostic” scrip, which 
detects a general error condition that does not have a 
specific solution supplied by another scrip. A set of 
about twenty diagnostic scrips is used to detect errors 
that are not resolved by the rest of the scrips. The first 
step of the escalation process is to initiate an update of 
the local database from the master database. In the 
most common case, the problem is a relatively new 
one that has been discovered and resolved since the 
last periodic update of the local database. For exam- 
ple, it might be a compatibility problem caused by 
installing a new version of a popular program. In this 
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case, the new scrip to resolve the problem 1s retrieved 
and applied. If, on the other hand, the problem is being 
encountered for the first time, the second step of the 
escalation process is to gather relevant information 
about the state of the system and the error and forward 
it electronically to the system administrator. This is 
typically much more complete information than the 
system administrator usually gets from end users. The 
administrator can then either reproduce the problem 
locally to determine its resolution, escalate the prob- 
lem to the vendor, or simply contact the user directly 
and resolve the problem through more traditional 
means. If the user is not local, the system administra- 
tor can initiate a “‘remote service”’ session, allowing 
the use of the remote interface module of the client to 
look at specific system state and apply problem reso- 
lutions. After understanding and resolving the prob- 
lem, either the system administrator or the vendor’s 
customer support staff initiates a process to add a new 
scrip to the master database. If the problem cannot be 
resolved and is escalated to the vendor’s engineering 
staff, it follows the same support process that it would 
for non-automated support. Once a resolution is avail- 
able, however, the system administrator or vendor 
adds a new scrip with the resolution to the master 
database. In any case, once the new scrip is in the 
database, the same problem is always resolved on all 
end user machines without further manual interven- 
tion. 


The software also provides a valuable service for 
administrators who are managing multiple end user 
sites by providing notification of all detection and res- 
olution activities. This can be used simply for account- 
ing purposes, so that the administrator can easily pro- 
vide activity reports to management demonstrating the 
ongoing value of the system administration staff. It 
can also be used for isolating trends that indicate a 
particularly troublesome hardware or software compo- 
nent at a particular site, leading to a replacement of 
that component and a resulting improvement in stabil- 
ity. Finally, it can be used to provide greater visibility 
to the system administrator. For example, the decision 
on when to upgrade an application to a new version 1s 
typically made on a fairly arbitrary basis today. How- 
ever, with this notification, the administrator is aware 
of the number of end user problems occurring that 
could be solved by such an upgrade. If the number of 
such problems is small, the upgrade can be postponed, 
but if it is large, the upgrade might be accelerated. 
Since most users rarely, if ever, report this kind of 
information, it is extremely valuable to system admin- 
istrators. Detailed reporting of problem detection and 
resolution activities also gives system administrators 
leverage when dealing with vendors’ customer support 
groups, since it minimizes the frustrating “‘finger- 
pointing” that typically goes on when problems occur 
in the multi-vendor computing environments that 
make up nearly all facilities. 
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This reporting mechanism 1s very configurable. 
li can be set up on a scrip-by-scrip basis to report 
immediately or collect the reports into a single log that 
Is sent on a periodic basis. An email interface provides 
the simplest, most convenient means of reporting. For 
system administrators who are maintaining a larger 
facility, a web-based interface can log reports by 
adding entries to a database. The administrator can 
then use a browser interface to peruse the database, 
spot trends, and manage user issues remotely. 


Scrips 


Each scrip consists of four parts: initialization, 
configuration, symptom, and solution. The symptom 
and solution parts have already been described; both 
parts are executable code that 1s loaded dynamically 
and run by the execution scheduler. The code for the 
symptom does a conclusive test to see whether the 
conditions for the problem actually exist. This is 
required since the events triggering a scrip may not 
completely define the situation for the problem. For 
example, the trigger may be an error dialog, but it may 
be necessary to look at some configuration data (in 
files or the registry) to determine whether the known 
problem 1s actually happening. The code for the solu- 
tion 1s the actual implementation of the solution. It can 
be extremely simple, for example, a solution that 
solely involves updating some configuration informa- 
tion. It can also be quite complex, for example, a solu- 
tion that has to try several different resolutions to see 
if any of them work, and must restart the system (and 
coordinate this process with the end user) after each 
try. 

Since the execution scheduler needs to set up the 
interface between scrips and primitives (and also 
between scrips and procedures) at runtime, the proce- 
dure interface across the boundaries of scrips must go 
through a thin interface layer that “‘interprets’’ the 
calls and returns. This has an impact on the perfor- 
mance of scrip execution; however, scrip execution 
happens at a low frequency and 1s not time critical, so 
the overall impact is negligible. On the other hand, 
one large benefit of this architecture is that symptoms 
and solutions for scrips can be implemented tn any 
language for which it is possible to implement this 
thin interface layer. Current implementations of serips 
use the “C” programming language, but this 1s mostly 
a matter of convenience, and interfaces for popular 
programming languages such as Perl (see [Wal00]) are 
being developed. 


The initialization part of a scrip has several func- 
tions. It registers the scrip with the execution sched- 
uler, specifying characteristics about the scrip such as 
other scrips with which if must be mutually exclusive 
(cannot run at the same time), and other scrips upon 
which it depends. Most scrips can work with a default 
configuration, but more complex ones can have spe- 
clalized requirements. Initialization of a scrip also 
defines the persistent state variables used by the scrip. 
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Finally, the tnitialization defines the conditions under 
which the scrip is triggered. This is done by specifying 
the conditions on a sequence of a few fundamental 
events. For example, a scrip can register itself to be 
triggered when a certain executable 1s run with a spe- 
cific command line, and then a dialog box 1s created 
by the resulting process with a given title. 


The timing and order of scrip initialization 1s left 
unspecified. For a client that needs to implement com- 
pietely dynamic configuration, the scrip initializations 
can be run sequentially as the client is starting. For a 
more conventional client (such as one used in desktop 
support), the scrip initializations can be run as a pre- 
processing step whenever the client 1s installed or the 
scrip database is updated. The results of the initializa- 
tion are saved in runtime tables that are read during 
client startup to provide fast initialization. 


The configuration part of a scrip allows the scrip 
author to create user-defined options that control the 
function of the scrip. This section specifies classes of 
input fields, such as radio buttons, check boxes, and 
text strings. Each input field has a default value and a 
persistent state variable associated with it. The user 
interface module presents the configuration informa- 
tion as XML, in a form that can be displayed and 
modified by a browser. Once this setup is complete, 
the client uses it to initialize the referenced persistent 
state variables. When the scrip runs, it uses the values 
of those variables to control its operation. In this way, 
the scrip author has great flexibility in allowing cus- 
tomization of the operation of the system. This facility 
also allows customization of “‘overall” features of the 
system, since the persistent state variables that a scrip 
configuration modifies are not limited in scope to vari- 
ables that affect a single scrip. 


There are actually four general types of scrips, 
although these types are all implemented the same 
way, so the categorization is a conceptual one rather 
than an operational one. The scrips that have been 
emphasized so far are those that solve a specific 
known problem with a well-defined and tested solu- 
tion. Another type of scrip is one that detects and 
resolves problems that arise as a result of end user 
operations, such as installing a new piece of hardware 
or software. Still another type of scrip is the “diagnos- 
tic’’ scrip previously mentioned that detects a general 
error condition that does not have a specific solution 
supplied by another scrip. The fourth type of scrip 1s 
the “preventive” one that performs routine adminis- 
tration and maintenance functions that can be auto- 
mated. Clearly, this last type of scrip is immediately 
useful to a system administrator, even in a system that 
is functioning perfectly. 


To simplify the management of scrips, and to 
allow them to be conceptually classified, a tree-struc- 
tured descriptive hierarchy 1s maintained. This 1s simi- 
lar in concept to the hierarchy that is maintained for 
information accessed by a search engine such as 
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Yahoo! (see [Yah00]). The hierarchy is expanded as 
needed when new scrips are developed, and a single 
scrip can be in more than one place in the hierarchy. 
The hierarchy is represented in the database contain- 
ing the scrips, and the user interface provides a facility 
for navigating the hierarchy to get to the configuration 
of individual scrips within it. Groups of scrips within 
a single branch of the hierarchy can be managed as a 
unit. For example, there are a group of scrips that 
implement basic functionality within the system, such 
as sending periodic logs and updating the client soft- 
ware to newer versions. These scrips are required for 
proper system operation, and are all part of the group 
of “‘system” scrips within the hierarchy. Implementing 
much of the system functionality using scrips makes it 
convenient to update the basic operating functions of 
the system without requiring software upgrades in the 
core software, resulting in a more reliable and flexible 
system. 


One of the key design goals in the architecture of 
the scrip database is to allow authoring of scrips at a 
multitude of facilities without any explicit coordina- 
tion. In this way, local expertise with certain types of 
problems can be captured and disseminated without 
having a central authority as a bottleneck. The name- 
space for persistent state, along with the self-organiz- 
ing features of the scrip initialization and the descrip- 
tive hierarchy, combine with the modular nature of the 
scrips themselves to make the promise of this widely 
distributed development a reality. 


Example 


A concrete example of a scrip will help to under- 
stand many facets of the system. This example scrip 
solves a problem that is taken from the direct experi- 
ence of one of the authors, a problem that recurred on 
a regular basis and was extremely annoying. 


The underlying cause of the problem lies in the 
shortcomings of the Microsoft Windows print server 
architecture. It would not arise in a purely Unix envi- 
ronment, but most system administrators recognize the 
necessity of managing systems with the popular Win- 
dows operating system, and certainly concur with the 
desire to quickly and efficiently handle the many prob- 
Jems it introduces. 


This particular problem manifests itself in this 
way: the facility has a Hewlett Packard LaserJet 4 
printer, driven by one of the file server systems run- 
ning Windows. Whenever a new system is installed, 
the printer is added as a network printer. Usually, 
everything works correctly, but occasionally someone 
prints a document and gets a dialog reporting that the 
document is “too complex” to print. However, if the 
document is transferred to another machine and 
printed from there, it prints with no problems. 


This problem is acutely frustrating for the end 
user, because there is no apparent difference in the two 
systems (Same printer driver, same document, same 
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operating system), yet the operation works correctly 
on one system but not on the other. In addition, the 
workaround of printing the document from another 
system is extremely inconvenient, because it disrupts 
the workflow of two people: the person trying to print 
the document and the person whose system must be 
used in order to print it. As a result, the person with 
the problem document usually ends up trying to 
“tweak”? it so that it isn’t “‘too complex” to print, 
resulting in wasted time and an inferior document. 


The cause of this problem is the fact that the 
printer drivers for the HP LaserJet 4 are installed with 
a default configuration of 2 MB of memory in the 
printer, even if the printer has more memory. This con- 
figuration must be manually changed to reflect the 
actual amount of memory in the printer, or else the 
printer driver will not be able to format the document 
to print because of this perceived lack of memory. In 
addition, the printer driver has a feature called “‘page 
protection” that must be turned on in order for the 
driver to accurately estimate the memory that 1s 
needed in order to print a particular document. Unfor- 
tunately, these two configuration changes must be 
made every time the printer driver is installed. Since 
end users may be installing these drivers themselves, it 
is difficult to set up a process that insures the applica- 
tion of these configuration changes. The situation 1s 
further complicated by the fact that a long period of 
time may elapse between the installation of the driver 
and the appearance of the problem. Justifiably, the 
reaction of the end user is that everything has worked 
perfectly for a long period of time, so there must not 
be any problem with the printer setup. 


This problem is ideal for solution with a simple 
scrip. Doing so provides a systematic way to represent 
“institutional” knowledge that is typically maintained 
informally by the system administration staff. The 
problem has a relatively simple method for recogniz- 
ing it, a fairly straightforward means to test for its 
existence, and a clear procedure for solving it. It is 
interesting to note that this problem, like many that are 
addressed by system administrators, is not technically 
a ‘“‘bug”’, in the sense that the software is working as 
designed. It is really more of a usability issue. 


The scrip for this example will be presented 
using ““C” as a programming language. The details to 
translate it to any particular programming language 
are fairly mechanical. The scrip initialization is shown 
in Listing 1. The initialization starts by registering the 
scrip with the execution scheduler, giving it a title and 
the default execution mode, which indicates that only 
one instance of the scrip should be run at a time. Then 
It sets up a trigger with the event detection mecha- 
nism, indicating that the scrip should be run whenever 
a dialog is created with the title ““HP LaserJet 4’ and 
the dialog text ““Document too complex”. Note that 
the first parameter for each primitive indicates the 
machine where the primitive should be run. In this 
case (and in all initialization), the primitives are run 
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on the current machine, which is indicated by the con- 
stant CUR. 


The symptom for this scrip is shown in Listing 2. 
The symptom gets the value of a specific registry key 
that contains the settings for the printer driver. If the 
registry key does not exist, then the scrip does not 
apply. It checks to see if the settings are the default 
ones for the driver, which are represented by a specific 
310-byte binary value. If so, then this scrip applies, 
because it is designed to solve the problem where the 
default value has not been updated to match the actual 
properties of the printer. Note that once again, the 
primitives are all run on the current system. Also note 
that the parameter to the scrip is a data structure 
describing information about the circumstances trig- 
gering the scrip, but in this case that information 1s not 
needed to determine the applicability of the scrip. 


The solution for this scrip is shown in Listing 3. 
The strategy for the solution is to assume that the 
printer driver settings are correct on the machine 
where the printer is installed, and copy the settings 
from that machine. It uses a registry key to get the 
name of the remote printcr, then extracts the name of 
the machine. It converts the name into the internal rep- 
resentation of a machine identifier, then gets the 


extern? void Init(void) 


{ 
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registry key for the printer settings. This is an example 
of executing a primitive on a remote machine. The 
same primitive 1s used to get the printer settings on the 
remote machine as was used to get them on the current 
machine in the symptom, but the first parameter indi- 
cates the remote machine rather than the current 
machine. Finally, the settings found on the remote 
machine are used to update the settings on the current 
machine. If at any point, the solution cannot proceed, 
it returns FALSE, indicating that it has failed. This 
failure will initiate the escalation process previously 
described. If the solution succeeds, then a successful 
problem solution will be logged as described previ- 
ously. 


Database Feasibility 


One important question about the system that 
naturally arises concerns the feasibility of creating and 
maintaining the database of scrips. We have done sig- 
nificant research to address this question. As men- 
tioned previously, the 80-20 rule indicates that the 
database size need only be about 20% of the total uni- 
verse of problems being addressed. Data presented in 
[All99, Etc98, IHSO0, ITS99, and Rea99] lead us to 
expect that 80% of all end user issues will be resolved 


/* Register the scrip with the execution scheduler. Use the 
default setup: “only one. instance of this Scrip at a timne.. */ 


RegisterScrip(CUR, 


"Update HP LaserJet 4 setup to correct Document Too Complex". 


STANDALONE) ; 


{*Set vp vthertkhipeer:£or the seri pa, 


SetDialogTrigger (CUR, 


o/ 


NULL, /* owning process */ 
MHP hacer det. a0 peat ke 
MMoetment too. complex" )s .f*° text 


Listing 1: Code for script initialization. 


extern BOOLEAN Symptom(TRIGINFO trig) 
{ 

char *setupVal; 

BOOLEAN exists, compare; 


/* Check (to. seeia ft “thevdriver 1s amstalied, and get setup. °7 
GetRegistryBinary(CUR, &setupVal, &exists, 
HHKEY - LOCAL MACHINE V\System\\\CurrentControlset\ \Control\ \"' 
'Praint\ \Printers\\He LaserJet -4\\Derault DevMode"™); 
(Tithe dna ver iene t anstarlody «the xseriap doesn t.dpply.. 67 


tf (lexis ts) -returm FALSE: 


j*"Sée-4f at 4s\ Still therdefault setup */ 
CheckMemoryEqual (CUR, &compare, setupVal, DEFAULTVAL, 310); 
/* lt the comparasone ts equal: the«serip applies: "7 


FevUri COmpare, 


Listing 2: Symptom code. 
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by this database. We have extensively characterized 
the Microsoft Knowledge Base (Mic00], focusing on 
the major products including operating systems, office 
productivity applications, groupware, and Intemet 
products. Using tools such as FAQs to identify the 
most frequently encountered problems, we found that 
a database with 2,500 to 3,000 entries should provide 
resolution of a large percentage of the recurring prob- 
lems encountered by end users in a Microsoft Win- 
dows environment. 


After the database is built, it needs to be main- 
tained on an ongoing basis. Figure 5 shows the num- 
ber of new entries as a function of time for the cate- 
gory “Windows 95” in [MicO0O]. Except for an 
8-month window around the operating system release, 
it shows about 25 new entries every month. Note that 
this 1s a pessimistic estimate, since not every entry in 
[Mic00] represents a problem, and even the “‘prob- 
lem” entries do not all represent solutions that require 
automation. By way of comparison, Figure 6 shows 
the number of new entries as a function of time for the 
category “Windows networks” in [Mic00]. This graph 
has no particular peak around a product release, but 
shows about 10 new entries every month. 


It seems reasonable to say that even an extremely 
pessimistic assumption for new problem incidence 


extern BOOLEAN Solution(void) 
{ 

char “ports 

char *machineName; 

char *setupVal; 

MACHINE mID; 

BOOLEAN exists; 
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will be less than 80 problems per month. Our initial 
work has shown that coding and testing a scrip takes 
less than four work days, so fewer than twenty full 
time scrip programmers can maintain the database. 


In a way, virus scanners serve as an “existence 
proof’’ for the feasibility of the scrip database. The 
effort involved in obtaining, analyzing, and producing 
a solution for a virus is not too different from the 
effort involved in understanding a problem and creat- 
ing a scrip for it. Current virus scanners have 
databases of about 45,000 viruses, so the effort 
involved in creating and maintaining the scrip 
database should be relatively small by comparison. 


Field Experience 


An initial version of the software described here 
is currently deployed and operational at a number of 
customer sites. It has been running at one site, a public 
relations firm with about 15 nodes, since February 
2000. Starting in March 2000, major functionality 
enhancements were released to that site on a weekly 
basis, and in April 2000 installations began at addi- 
tional customer sites. As of the time of this writing 
(September 2000), the software is running at ten cus- 
tomer sites. 


f°-9Ger the-machine where the printer i1s.instal led: 7 


GetRegistryString(CUR, &port, &exists, 


"HKEY LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\" 
UPrintv\erintersV\HE Laserjet. 4\vPone™); 


ita (existe). retarn ALOE. 


at VOC pore, f=-n\ Ne): i) | Cporuli tah) return FALSE: 


GetSubstring(CUR, &machineName, port, 2, 
GetMachineID(CUR, mID, machineName) ; 


Nee 


/* Ceti the printer setup. from that machine... *7/ 

GetRegistryBinary(mID, &setupVal, &exists, 
"HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\" 
'Prant\\Priters\\HP bagerJet. 4\\Default DeyMode)< 


if  Clexiets) return FALSE: 


j/* Use that printer setup: on. this machine. */ 


SetRegistryBinary(CUR, &setupVal, 


"HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\" 
"Print\\Printers\\HP LaserJet 4\\Default DevMode") ; 
/? Notify end user of requared action. “7 


NotifyUser (CUR, 


"Your printer setup has been updated to correct the " 
"’Document too complex’ problem. Please try printing " 


"the document again."); 
return TRUE; 


Listing 3: Solution code. 
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The customer feedback has been very positive. 
None of the customers have reported any problems or 
performance degradation with the software, and the 
software has proven its value in identifying major 
issues at every single one of the sites. The system 
administrator at one site was pleased to report that the 
logs have provided information on problems that 
would otherwise go unreported. At another site, the 
logs indicated a serious problem that was preventing 
the virus scanner from running, which had gone unde- 
tected for months! 


The feedback from customer sites has also 
proved very valuable to our development team. The 
client checks every error return and uses its 
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potentially serious, bugs that were not found during 
testing. In early versions of the client, the problem res- 
olution capabilities have been limited, so every event 
was escalated through the reporting mechanism. These 
reports have guided us in our choice of scrips to 
implement for the initial database, so we are confident 
that we are populating the database with the most use- 
ful scrips. One reassuring “‘reality check”’ is the obser- 
vation that the problems that occur at all ten customer 
sites are remarkably similar, even though the busi- 
nesses themselves are considerably different. 


Open Source Development 


The scrip database lends itself very naturally to 
an open source development methodology. The initial 
commitment to the development of a scrip is small 


Win 95 Knowledge Base 
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Figure 5: New Problem Incidence in Windows 95. 
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Figure 6: New Problem Incidence in Windows Networking. 
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since it only takes a few days, and the payback is 
immediate since it solves a persistent and annoying 
problem in a permanent way. Scrips are likely to be 
useful at more than one site, so the code that goes 
back into the open source pool will be immediately 
beneficial to the user community at large. This 
approach also helps to populate the database with the 
most important scrips first, since the motivation will 
be highest to solve the most severe problems. The 
Open source methodology also adds a measure of 
security, since open review of the scrips will deter any 
malicious activity. A more subtle benefit arises from 
the fact that scrip development has several distinct 
phases, including problem resolution, solution defini- 
tion, coding, and testing. These phases require differ- 
ent skills and can be completed most efficiently in a 
cooperative group environment. For all these reasons, 
we are using an open source methodology for scrip 
development. 


Future Directions 


Although this system is primarily focused on 
reducing system administration headaches by stream- 
lining end user support, it lends itself naturally to sev- 
eral other uses that can also make an administrator’s 
life easier. As previously described, we have written 
scrips both for bugs and for usability issues that are 
frequently encountered by end users and are confusing 
enough to generate calls for help. In addition, we have 
written scrips to automate common administration and 
maintenance functions that help to keep a properly 
functioning system running smoothly. We also see 
great utility in using scrips to distribute security 
patches, since the conditions for their use and the pro- 
cedures for their application are often complex enough 
to warrant code for their implementation. 


The technology is flexible and useful enough to 
apply to the next generation of “pervasive” comput- 
ing devices. As the load on system administrators 
expands to include support of mobile and other 
embedded computing platforms, the need for auto- 
mated support is only going to become more critical. 
The lightweight, standards-based client is ideally 
suited fer these environments as well as the applica- 
tions that support them. 


Summary 


The HandsFree Networks support automation 
tool described in this paper relieves system adminis- 
trators of the burden of dealing with mundane issues 
by automating the resolution of many common recur- 
ring problems. It uses a standards-based extensible 
architecture to provide a database of solutions that is 
applied without manual intervention. An open source 
development methodology for the database and a 
built-in incremental database update ensures that the 
solutions will keep up with the introduction of new 
products and the corresponding appearance of new 
issues. 
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ABSTRACT 


FTP Mirror Tracker’ is a software package (written in Perl and C++) that enables transparent, 
user-controlled redirection to the nearest anonymous FTP mirror sites that are exact replicas of the 
original source. This redirection can be achieved by using a Web Cache server or by making 
HTTP requests to the FTP Mirror Tracker directly. The Mirror Tracker also has intemal URN 
support and can be used as a URN resolver for FTP requests. Underlying the system is a MySQL 
database recording FTP mirror site details. In this report we explain how this database is 
constructed, and show how it may be used — directly by end users, and under the policy based 
control of Web Cache and mirror service administrators. 


Introduction 


Although FTP traffic passing through the modern 
Internet only accounts for a small fraction of request 
transactions, its bandwidth utilization is significant. 
For example, FTP accounted for between 7% and 11% 
of the incoming traffic on the JANET? network’s links 
to the United States for every month in 1999. 


There is a long standing Internet convention that 
sites which are particularly large (e.g., operating sys- 
tem distributions) or popular (e.g., the Starr Report) 
will be widely replicated — usually by volunteer effort. 
The replication process, which typically takes place on 
a daily basis, is usually referred to as mirroring. Mir- 
roring software exists for replicating Web (HTTP), 
FTP and (by prior arrangement) arbitrary content, e.g., 
GNU weet [2], mirror [3] and rsync [4]. In recent 
years some formalization of this role has taken place, 
e.g., with the establishment of the UK Mirror Service 
[5] for JANET users, and the AARNet2 Mirror 
Archive [6] for Australian academic and research 
users. Localizing what might well be international 
(and chargeable) traffic to geographically and/or topo- 
logically nearby mirror sites is a challenging task. 


A number of approaches to brute-force indexing 
of the FTP namespace have been attempted, notably 
the Archie [7] system from Bunyip, CNET’s Share- 
ware.com [8] and Lycos/FAST FTP Search [9]. The 
user interfaces for searching these systems typically 
allow the end user to supply full or partial filename 
details or regular expressions to match, and retum a 
list of the UR Ls where files matching the search crite- 
ria can be found. Because of the difficulty in knowing 
where on the Internet 1s the best topological/policy 
match for a client, only minimal attempts have been 
made to provide tailored output on a per-user basis. 


‘http://squid.itep.ru/ mirrored at http://wwwcache.ja.net/ 
mirrors/MirrorTracker/ 

2JANET [1] is the UK’s Higher Education and Research 
Network 
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The FTP Mirror Tracker Design in Brief 


The core of the FTP Mirror Tracker software is a 
robot which traverses through a list of anonymous 
FTP servers which it has been told to visit, connects to 
each of them in tum, fetches their public directory tree 
content (using the FTP Is -IR command), analyses it 
and creates a unique identifier (actually an MDS [10] 
digest value) for the content of each directory and the 
tree of the directories (using a summarizer program). 
These identifiers and the FTP URL paths they refer to 
are stored in a MySQL database [11] using the Perl 
[12] DBI database interface [13]. In turn, the MySQL 
database is accessed by a variety of programs which 
collectively form the “‘user interface” to the system. 


The processed data created by the Mirror Tracker 
summariser is also made available for use by other 
services, e.g., for sharing with other Mirror Tracker 
servers. By default each Mirror Tracker maintains 
copies not only of the data for the Internet domains it 
is responsible for, but also the data for the domains 
which are indexed by other Mirror Trackers. A ‘‘root”’ 
Mirror Tracker has been established in Moscow, Rus- 
sia to help bootstrap this process. 


From a user standpoint, the basic operation of the 
FTP Mirror Tracker is to take a URL and retum a list 
of the alternative URLs where this same material may 
be found — subject to search criteria such as the top 
level domain(s) required in the search results. This is 
done by finding the unique identifier associated with 
the requested URLs, and checking to see whether any 
other entries in the table(s) in question have the same 
unique identifier. 


We will use the phrase “unique identifier” rather 
than ““MDS digest value” throughout this paper, since 
there is no actual requirement that MD5 digests be 
used as the resource collection identifier. The unique- 
ness and location independence of our identifiers also 
makes them attractive as a global naming system, 
though it should be noted that the way they are 
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calculated means that they are not persistent over time. 
This means that they are not truly suitable for use as 
URNs. However, since there 1s very little practical 
deployment experience with URNs, we have chosen to 
ignore this problem for the time being. 


The Gory Details 


The FTP Mirror Tracker architecture consists of 
the following components: 

A robot which collects directory listing data 
from the FTP servers it has been configured to 
track. 

e A summarizer which creates MDS digests of 
the directory listings. 

e A digest exchanger, for sharing the digests with 
Other servers. 

e A back end database — currently MySQL. 

e Various frontend programs. 


We will describe in short the design of each of 
these components, and how they have been imple- 
mented. 


The Mirror Tracker robot 


The function of this component is to gather raw 
directory listings from FTP servers for processing by 
the summarizer. The robot has been implemented as 
two Perl programs — robot, which parses the list of the 
FTP servers for each domain begin indexed and forks 
a second Perl script, ftp_list, to actually fetch the direc- 
tory listing for the server. 


If it was able to start an anonymous FTP session 
with the target server, ftp list proceeds to check 
whether there is an Is-IR.gz file in the root or /pub 
directories on the server. If one that it 1s reasonably 
fresh (and not empty) 1s present, this 1s fetched, other- 
wise the |s -IR command is issued from the top level 
directory of the anonymous FTP server in order to 
produce a recursive directory listing. 


The summarizer 


This is the heart of the FTP Mirror Tracker. Its 
job is to parse the directory listings of the FTP servers 
which are being tracked (fetched by the robot), ana- 
lyze them, and create MDS digests based on this anal- 
ysis. The summarizer is implemented by a C++ pro- 
gram, createdigest, which reads in the raw directory 
listings and generates a list of MD5 digest values and 
URLs on a per-directory basis. 






1 _ ftp://ftp.xfree86.org 

2 _ ftp://ftp.x free86.org/pub/ 

3. ftp://ftp.x free 86.org/pub/XFree86/ 

4 _ ftp://ftp.xfree86.org/pub/XFree86/4.0.1/ 
5 ftp://ftp.gamma.ru/.3/XFree86/4.0.1/ 





ftp://ftp.gamma.ru/.3/XFree86/4 .0.1/binaries/ 
ftp://ftp.gamma.ru/.3/XFree86/4.0. 1/binaries/Linux-1x86-glibc21/ 


Novikov & Hamilton 


It might seem sensible to use all of the informa- 
tion from the directory listing output when creating 
the digest of it (1.e., permission, node, owner, group, 
size, date and the filename). Unfortunately most of 
these values can be changed during the mirroring pro- 
cess. In practice it seems that we can get by using only 
the file size and name (though as noted, some mirrors 
will compress files — we exclude these), plus the file 
type (e.g., plain file, directory or link). 


We will note in passing that a better approach to 
persistence would be to calculate unique identifiers 
based on the contents of the files themselves, rather 
than the directory listing metadata. A program has 
been provided with the FTP Mirror Tracker distribu- 
tion to let FIP server administrators calculate MD5 
digest values for each file in a given directory hierar- 
chy. However, this 1s very much a “‘future’’, since it 
would have to be widely deployed in order to be gen- 
erally useful. 


total 821 
=r+xr-xr-x % root £tp 62163 Jan 25 19:43 compress 
r-xr-xr-x @ root ftp 168240 Jan 25 19:38 date 
GC KE- xix Me vroot ttp: 106/52 Jan’ 25.(19338- gzip 
°LoxE= XE 2 root ftp 186048: dain. -25, 19-37 1s 
PoE RP ee TOOt ftp 270292 Jan. 25 19 38 tar 


Listing 1: Typical output of the Is -IR command. 


So, in order to produce the digest of a directory, 
we take the size and name of each of the files within it 
and concatenate them, e.g., in the example above we 
are left with: 

62163compress168240date106752g21p1868481s270232tar 


We then create a hexadecimal representation of 
the MDS5 digest value for this string and assign it to 
the URL of the original directory on the FTP server. In 
this way we can create a relatively small unique iden- 
tifier for the directory contents rather than for the files 
themselves. Finally, we add this text to the parent 
directory listing string, giving an identifier for the con- 
tents of the complete directory structure on the FTP 
server. 

To understand why we need to add the unique 
identifier of the subdirectories to the string representa- 
tion of the parent directory, let’s take a simple exam- 
ple. Let’s assume that someone wants to fetch the lat- 
est XFree86 for Linux runing on x86 with glibc 2.1 
from the nearest server. This person starts to look for 
an exact replica nearby; see Table 1. If we hadn’t 


no exact replicas 
no exact replicas 

still no exact replicas 

found some! so he shifts to 
the nearest one 








Table 1: Search sequence for exactly replica. 
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added information about the subdirectories to the 
directory information, we couldn’t be sure that the 
Linux-1x86-glibc21 directory is located somewhere on 
the mirror server under the 4.0.1 directory. 


The Database Back End 


We use a simple database design, with a dedi- 
cated database for the FTP Mirror Tracker data, and 
separate tables within this for each of the digest and 
link collections for each of the top level domains 
being tracked. 


We are able to take advantage of the MySQL 
“load data” feature to read in the digest and link files 
directly, with MySQL automatically creating a 
database row in the appropriate table for each line of 
these files. The actual database manipulation is done 
using Perl, with DBI for database access. 

The Digest Exchange 

The original files are compressed and are moved 
to a directory which 1s accessible through the WWW, 
so that they can be shared with other Mirror Trackers. 

Using the method described above we parse, for 
example, 100 MB of compressed listings from FTP 
servers (representing almost all of the anonymous FTP 


Fite Edit View Go Communicator - ae . 


Search _Newcape i, ‘ a 


2 > 3 ume ; 


7 Back Foard Rete 


2 wa” Bookmarks A 


We are tracking »p tacker 


Other possible locatiu 


ee ———————— on 
tp : /AEtp.xfree86. org/pub/iFree86/3. » 
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servers in Germany), create a 50 MB digest file and a 
25 MB links file for feeding into the database engine. 
After compression, these files are 7 Mb and 2 Mb 
respectively — small enough to be shared easily via the 
Www. 


Frontend Programs 


We provide a simple CGI program (written in 
Perl and C) which lets the end user query the FTP 
Murror Tracker for specific URLs. This can be inter- 
acted with directly by the user, or linked to by content 
providers. There is also a third mode of operation 
which we will call Mirror Tracker on Demand. 


Mirror Tracker on Demand is a JavaScript com- 
mand which can be saved as a bookmark or (for exam- 
ple) placed on the Netscape Communicator Personal 
Toolbar (shown wrapped): 

javascript: Location. hret— http //, 
tracker.foo.bar//cgi-bin/ 
tracker?uri=" 
escape(window.location) ; 


Pressing this button will cause the current URL 
to be sent for resolution by the FTP Mirror Tracker, 
and the results of the Mirror Tracker search to be 





om 2 B 


| ee What's Related 


| If you still want to use the Orit, servers HOUR the gtobe. 


Pre. 


| (no mathematical sir aw. aublic.xfree8G.o 
uliGe eet 


= 


yetserve Chal 


fio. gamma iu SOAF rees6/3,3.5/ 


All files at these links have the same size but creation date may differs (this is acommon problem with the mirror sites). 


Software used for this redirection is in beta stage. It is provided in attempt to be useful but no guarantee can be given. 
If you have any questions or comments please e-mail me. Alexei Novikov 





Figure 1: Typical browser window with the output from the WWW frontend. “Mirror Tracker”’ button is zoomed. 
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presented in the browser’s main window — as shown in 
Figure 1. 


Although these user interfaces are very simple, 
they are also extremely powerful — instead of using 
search engines periodically to find out who 1s mirror- 
ing which sites, and trying to figure out which mirror 
sites are fresh and which are stale, content providers 
can have just one link on their site to the nearest FTP 
Mirror Tracker. The Mirror Tracker software will pro- 
vide users with the list of the sites which have fresh 
and accurate replicas, optionally located in the same 
domain as the user. 


Content providers only have to register with their 
nearest FTP Mirror Tracker server and put a link to it 
on their Web page. The link will look something like 
this (show here with lines wrapped for display pur- 
poses); 

Ca Dref="http:// squad -ttrep.< ru/ 
cgi -bin/ tracker? url=YOUR URL"> 
YOUR PACKAGE</ a> 


For example, the following link would yield all of the 
mirror sites known for the XFree86 3.3.6 release 
(shown wrapped): 

“a hret="http:// squid 1tep.ru/ 
cpgi-bin/tracker?url=ftp:// 
ftp.xfree86.org/pub/XFree86/ 

35356) "23.3.6 Versi on;Mirrore </a> 


Web Cache and Mirror Service Integration 


Architecture for Cache/Mirror Cooperation 


Whilst there is no standard network protocol for 
building and interacting with mirror services (rsync 1s 
probably the closest thing we have to this), there are a 
variety of ways in which Web Caching systems can be 
made to interoperate with each other — and in turn 
with Mirror services. In addition to the core proxy 
HTTP service, most Web Cache products also support 
interoperability via the Internet Cache Protocol [14], 
and the freeware Squid Web Cache [15] supports the 
new Cache Digest protocol [16]. Other cache coopera- 
tion protocols exist, such as the HyperText Caching 
Protocol [17], but these have yet to be as widely 
deployed and will not be further considered in this 
paper. 

Since we do not wish to build our own Web 
Cache server, we have opted to use Squid for proxy 
HTTP services and graft on our own ICP server and 
Cache Digest generator — these are described in 
greater detail below. 


Many Web Cache packages provide a URL 
rewriting facility, although some of these are limited 
to taking in a list of URLs to rewrite and the URLs to 
rewrite them to — or a list of URLs to block access to. 
So we can redirect desired URLs to the server with the 
FTP Mirror Tracker resolver. 

Squid’s approach, the “redirector” program, is 
particularly well suited to use with the FTP Mirror 
Tracker, since (when it is enabled) URLs are passed to 
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an external program for analysis and potential rewrit- 
ing. For performance reasons (the external program 
doing this may block whilst waiting for an operation 
to complete) Squid forks a large number of redirector 
programs which run continuously in parallel to the 
main Squid server process. 


Use of the Internet Cache Protocol (ICP) for Cache 
Cooperation 


The Internet Cache Protocol allows a Web Cache 
(or any other program which is interested in talking to 
Web Caches) to query another Web Cache as to its 
contents. Each ICP query or response takes the form 
of a UDP packet (with all fields in “network” or 
““‘big-endian” order), consisting of a twenty byte 
header with fields and a variable length payload. Typi- 
cally the payload consists simply of a URL. 


ICP is, for all its problems, the most widely 
implemented Web Cache cooperation protocol, avail- 
able in most freeware and commercial caching prod- 
ucts. Consequently, we decided to support it as an 
access method for the FTP Mirror Tracker too. Rather 
than modify Squid’s own internal ICP server, which is 
very closely tied to the rest of Squid, a Perl module 
WebCache::ICP [18] was written to encapsulate the 
details of ICP packet processing within a simple 
object-oriented interface. This gives us an easy way to 
provide an ICP service which returns whatever ICP 
response we like on a given ICP request. 


The ICP server we use with the FTP Murror 
Tracker queries of the Mirror Tracker database to 
determine whether the Mirror Tracker should be vis- 
ited for a given URL in the ICP request packet. In the 
case of downstream caches which are running Squid, 
we can reduce the amount of ICP traffic associated 
with the peering by using the “cache _peer_access”’ 
Access Control List, e.g., to specify that only FTP 
URLs should be sent to this peer. 


Use of Cache Digests for Cache Cooperation 


Cache Digests (implemented in Squid since ver- 
sion |.2beta) offer a completely different approach to 
sharing information about cached objects, but have yet 
to be widely deployed in commercial Web Cache 
products. We decided to include them in the FTP Mir- 
ror Tracker project because of their widespread use 
with Squid, but the reader should note that they are 
still classified as experimental and subject to change. 


Whereas ICP requires request and response UDP 
packets to be exchanged, Cache Digests work on the 
principle that the Web Cache builds a summary of its 
contents. This summary takes the form of a 128 byte 
header, followed by a bit array. This summary is then 
made available to other Web Caches via the normal 
proxy HTTP interface, and these are then in a position 
to do a local (Digest) lookup when trying to determine 
whether any of their peers has a requested URL. 


In creating a Cache Digest populated from the 
Mirror Tracker database, we have the problem that 
there is no way to register whole URLs — the Mirror 
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Tracker database currently does not contain this infor- 
mation. Fuller Cache Digests could, however, be con- 
structed usint the raw directory listing information 
which is processed in order to create the Mirror 
Tracker database. 


Uniform Resource Names (URNs) 


Introduction to URNs 


URNSs [19] are being developed by the Internet 
Engineering Task Force as an alternative naming 
scheme, complementary to the existing URL. Whereas 
URLs essentially encode a ‘‘recipe”’ of instructions to 
be followed when fetching a given copy of a resource, 
URNs are required to be: 

e Location independent, so that the instructions 
for reaching the resource are de-coupled from 
the name by which the resource 1s cited, e.g., in 
Web documents. 

e Persistent, so that the assignment of a URN toa 
resource effectively acts as a guarantee that the 
resource will continue to be accessible by this 
name indefinitely and will never be replaced by 
another resource which has the same URN. 

e Compatible with existing name spaces such as 
ISSN and ISBN numbers. 


In practice this is intended to be done by defining 
a number of “namespaces’’, each of which will have 
its own procedures for things like registration and 
management. URNs themselves [20] are simply the 
concatenation of the string “‘urn’’, a centrally assigned 
(by the Internet Corporation for Assigned Names and 
Numbers [21] ) identifier for the namespace in ques- 
tion, and then a ““Namespace Specific String” — sepa- 
rated by colons. That 1s: 
urm:Namespace Identifier:Namespace Specific String 


The authority responsible for each namespace 
[22] is free to subdivide the Namespace Specific 
String component of their URNs as they see fit — sub- 
ject to character set encoding requirements which are 
designed to ensure that URNs may be used on the 
widest possible range of devices. The Namespace 
Identifier itself 1s chosen out of the set of upper and 
lower case letters, numbers and the hyphen character -, 
though the first character may not be a hyphen. The 
Namespace Specific String has a slightly larger vocab- 
ulary, which includes some punctuation characters. 


URNs and the FTP Mirror Tracker 


As noted above, the Mirror Tracker unique iden- 
tifiers are subject to change over time. This means that 
they cannot be true URNs as these are defined by the 
IETF. However, since there is still a dearth of true 
URNs for people to experiment with, we shall ignore 
this limitation for the moment. 


We supply a Squid compatible ‘“‘N2L” [23] pro- 
gram with the FTP Mirror Tracker, using the Mirror 
Tracker “unique identifiers” as URNs and the experi- 
mental x-tracker Namespace Identifier. An example of 
a Mirror Tracker URN would be: 
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urn:x-tracker:57ce08343306laab97c9c2b63759eF2F 


This is the unique identifier for all of the copies 
of the directory listing which can also be referred to 
by the URLs (as in the summarizer examples above): 

ftp://ftp.xfree86.org/pub/XFree86/4.0/ 


ftp://netserv2.chg.ru/pub/XFree86/4.0/ 
ftp://ftp.gamma.ru/ .3/XFree86/4.0/ 


ftp://caramba.cs.tu-berlin.de/pub/X/XFree86/4.0/ 


ftp://wizard.freesoftware.com/.0/XFree86/4.0/ 
ftp://ftp.linux.tucows.com/pub/XFree86/4.0/ 


Unfortunately, since even Netscape must be 
explicitly configured to talk to a proxy for URN reso- 
lution, we cannot simply deploy support for URNs as 
part of (for example) the JANET Web Cache Service 
and have them immediately be available to the end 
user. Each of the sites which connects to the service 
would need to enable URN support in their own users’ 
browsers too. 


Summary 


The FTP Mirror Tracker enables transparent, 
user-controlled redirection to the nearest FTP mirror 
sites which are exact replicas of the original source. 
The redirection can be achieved by using a Web Cache 
server or by making an HTTP request to the FTP Mir- 
ror Tracker directly. The Mirror Tracker has internal 
URN support and can be used as a URN resolver for 
FTP requests. 


During the course of this work we have produced 
a variety of tools and documents which may useful for 
other purposes above and beyond the FTP Mirror 
Tracker itself, e.g., the ICP and Cache Digest Perl 
modules, the Cache Digests specification, and the FTP 
robot. The software, databases, and documentation 
associated with the project are available for download 
via its homepage http://squid.itep.ru/. The changes 
which we made to Squid have been folded into the 
mainstream Squid 2.3 distribution, released in January 
2000. 
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Management and Deployment System 
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ABSTRACT 


Many administrators use a central software repository because managing distributed 
packages 1s difficult. Deployme is our solution to manage the package update lifecycle across a 


large number of independently configured hosts. 


Deployme is highly flexible and has been extended to handle many different types of 
packages. Deployme packages include standard UNIX tools, local applications, web site content, 
and voice site content. Most packages require fast, frequent deployment. 


Deployme has a web-based user interface that allows less technical users to deploy on their 
own. Deployme also restarts appropriate server processes, a feature which was much more 


difficult than we expected. 


We discuss other lessons leamed from the first implementation and planned improvements 


for the future. 


Introduction 


Using small, independent software packages is a 
common approach to managing software. Many tools 
have been developed to address package management 
including Depot [Manheimer]990] and LUDE [Dage- 
nais1 993]. 


The package philosophy behind these tools is 
easily extended to rapidly changing content. However, 
we found these systems to be incomplete because they 
only cover one or two stages of the package update 
lifecycle. 


Similar to the software distribution cycle out- 
lined in [Furlani1996], we have divided the package 
update lifecycle into four steps: 

1. Create the package. 

2. Distribute the package to designated hosts. 

3. Install and activate the package on each remote 
host. 

4. Delete old, unused packages from remote hosts. 


Deployme automates package creation and elimi- 
nates the need for a release engineer to manually pack- 
age data. (As you may have guessed, ‘“‘Deployme”’ is 
one of many tools derived from Tellme’s corporate 
moniker.) Deployme exports source data from CVS 
[Cederqvist!993] and then builds executables and 
other generated files. 


The system also handles installation, activation, 
and removal of packages, completing the package life- 
cycle. Deployme automates most of the traditional 
release engineer role. 


The original design was intended to manage and 
deploy web site and voice site content. We have since 
found the system to be general enough to be used for 
local applications, common tool management, and 
other types of content. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


With Deployme, our producers and developers 
manage their own releases. A straightforward web 
user interface allows the development team to update 
the Tellme service without the intervention of our net- 
work operations team. 


Motivation 


Our web-based content management system 
before Deployme was difficult to maintain, but was 
easier to use than we expected. This success led us to 
make a simple user interface a major Deployme goal. 


As applications passed quality assurance, content 
developers checked their changes into a CVS reposi- 
tory. Once all applications had reached a known good 
state, the tree could be released. 


To perform a release, a developer would go to a 
web page with a single button on it. The button started 
a shell script that checked out the CVS content tree 
and copied the data to the production server. 


Platform releases were similar. Again, we built a 
web page with a single button that kicked off a shell 
script. The script performed a CVS pull, ran the build, 
and copied the release to the production server. As 
more servers arrived, we added them to the list that 
received completed builds. 


The benefit of the release scripts was that they 
took much of the manual work out of releases and 
freed up developers to do more constructive tasks. The 
scripts also significantly reduced developer stress, as 
platform releases became an everyday occurrence 
instead of a frightening event. Finally, they allowed 
any developer to be appointed release engineer, 
because the only process involved was filling out the 
form on the web page. 


The script proved increasingly difficult to modify 
as we increased the number of build targets and 
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machines. It also could not support features such as 
rollback. If we wanted to push the same build to a new 
machine using the script, we had to wait while the 
script recompiled the platform from scratch. It became 
obvious that our inefficient distribution script could 
not scale. 


At this point, we seriously considered having our 
production servers attach to a storage area network 
(SAN) instead of distributing software packages to 
local storage. We decided against this for several rea- 
sons. 


Distributing packages to local disks still provides 
high availability, but it is cheaper and more flexible. A 
centralized file repository does not span across cities 
and continents without significant cost in time, hard- 
ware, and complexity. Also, most centralized solutions 
would lock us to one vendor. Our failures are 
restricted to individual machines, not an entire storage 
array. Even with high-availability components, SANs 
still act as a single point of failure. 


So, to replace the unscalable release script, we 
wrote a new script to perform releases. This script was 
the first implementation of our distributed package 
architecture. However, it only performed distribution 
and activation of packages. It had no web interface, 
requiring a network operations engineer to serve as 
gatekeeper for all releases of content, platform, tools, 
and configuration files. 


Updates to the production system became very 
infrequent. Developers plaintively asked, “‘Why can’t 
we bring back the button?” 


Goals 


Deployme’s mission is to provide a central sys- 
tem for tracking the entire lifecycle of software pack- 
ages. We established the following goals to judge the 
success of the project: 

e Support a wide audience. Users who are less 
inclined to system administration tasks should 
be able to create and deploy packages. A 
human gatekeeper will impede updates. 
Robustness. With a wide audience using the 
system, the system must never activate a pack- 
age without verifying that it was distributed 
successfully. Detailed audit logs and notifica- 
tion must be included to inform our Network 
Operations team of system changes. 

e Augment the development process. Packaging 
should not be a complex task, but instead it 
should be seamlessly integrated into existing 
processes. 

Flexible destinations. Packages should be able 
to be deployed to various destinations such as 
development systems, testing systems, and pro- 
duction systems. 

Efficient use of network bandwidth. When large 
groups of servers are located at remote data 
centers, Deployme should only push bits over 
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the slow link once and distribute where there is 
good bandwidth. 

Quick pushes. The system should allow, but not 
enforce, staging of a package for testing. If a 
“hot fix” (e.g., a critical bug fix or a press 
release for the web site) must be pushed imme- 
diately, it should be possible to bypass staging 
servers. 

Seamless activation. End users (customers) 
should never be aware of a new package being 
pushed (e.g., Web site links should never be 
broken). If a server binary is being pushed, the 
server process should be gracefully restarted. 

e Rollback. It should be possible to return to a 
known good state quickly and robustly. 
Scalability. Deployme needs to be able to han- 
dle hundreds of modules, hundreds of servers, 
and thousands of packages. 


Non-Goals 


We intentionally chose not to address several 
issues in the first implementation. 

e No local package management. Enough robust 
single-server package managers have been 
developed in the past that we should not invent 
a new one. 

e No dependencies. Our aggressive schedule 

would not leave us enough time to develop 
dependency tracking properly. We felt we could 
achieve all of the goals above without depen- 
dency tracking. 
No fine-grained operations control. While we 
chose to integrate certain operations functional- 
ity into Deployme, we decided against tuming 
Deployme into a “control panel” for control- 
ling remote processes. 


Previous Work 


There is much previously published work on 
package management and deployment tools. Many 
tools, such as Depot [Manheimer1990], take advan- 
tage of transparent remote network file system access 
such as NFS [Sandberg1985] or AFS [Howard 1988]. 
A system that contains a centralized database contain- 
ing server configuration information is presented in 
[Finke 1997]. Microsoft has an architectural discussion 
of their corporate website publishing tool in 
[Moore1 999]. 


Various methods for remote execution and server 
process restarting have been presented in tools such as 
Igor [Pierce 1996] and Synctree [Lockard 1998]. 


As stated in the previous section, Deployme does 
not address local package management directly. 
Instead, it calls routines to ‘“‘activate’” and “‘deacti- 
vate” packages on a machine. In our environment, we 
use GNU Stow [Glickstein 1996] for this purpose. 


Design and Implementation 


The biggest problem with the old content release 
process was that all content was released at once. 
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Every tree push was potentially catastrophic. If any 
one application had some code checked in that had not 
been fully tested, we would have to roll back the entire 
tree. We learned that a big tree of content 1s more dif- 
ficult to test than a big tree of code. 


We decided to rearrange the entire content sys- 
tem in order to make it possible to release one applica- 
tion at a time using Deployme. 


Deployme ss written entirely in Perl5 
[Wall2000]. We chose Perl because of our familiarity 
with it, its straightforward database integration, and 
our desire to attempt a complex, structured project in 
the Perl language. 


Deployme has a simple three-tier architecture: 
the user interface, the rules logic, and the database. 


User Interface 


The Deployme interface is a set of web pages 
that display a list of available products and a list of 
potential target servers. There are also information 
pages that show which packages are active on a given 
server. 


Each product is called a “module”, after the 
CVS term. A product could be a collection of web 
pages, or the source required to build an executable. 


Each release of a module is called a “‘tag”’, again 
after the CVS term. A tag corresponds to a physical 
package on disk. The package is assembled by 
requesting that the version control system write out 
the tagged version of the module. 


| > Click for recent Jobs < | 


Module Deploymant: 


Select a view (currently viewing: vui): 
Www Developer 
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All Modutes | 
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fats 





Pruductian Content 
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vei eddy = lackiack =|) Gl [vurbleckjack-20000804-1302RELEASE—=~=~S~* 


Qa [vu “horascopes- -20000906-1949 RELEASE 
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The central assumption of Deployme 1s that the 
contents of a tag never change. This 1s stricter than the 
version control sense of ‘‘tag’’, since most version 
control systems allow users to move a tag forward or 
backward on a given file’s timeline. 


Because of this assumption, Deployme’s tag 
namespace is a superset of the version control tag 
namespace. For example, the CVS tag “engine-1”’ can 
be tagged in Deployme as ‘“engine-1.DEBUG. 
1386.solaris.5_6’’ or “engine-1.RELEASE.sparc.solanis. 
5_7’. The source code is the same, but the resulting 
package is compiled with the appropriate build flags 
and architecture. 


By assuming that packages never change, we can 
skip crosschecking between servers to establish the 
version of a package. This also means that the slight- 
est modification requires a new tag. 


To start a Deployme Job, the user first picks a set 
of tags. The user is restricted to only selecting one tag 
for each module, because Deployme does not allow 
two versions of the same module to be active at the 
same time. Then, the user picks a set of servers. 
Deployme checks to ensure that the request does not 
break any rules, and then executes a job fulfilling the 
request. 


To rollback a module, the user selects a previous 
tag and follows the same procedure. 


Rules Logic 


The middle tier comprises the rules logic that 
creates and executes a deployment Job. 








= 


Select 





| [™ contentO1.tellme.com 


Push modules 


Do NOT LINK modules J" 
Dc NOT RESTART modules [~ 





Figure 1: The Deployme web interface. 
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Deployme’s logic falls into two categories: job 
creation logic and job execution logic. The former 
uses a set of rules to determine what tasks should be 
done. The latter uses the database state tables to carry 
out those tasks. 


Job creation expands the list of packages and 
destinations into a series of tasks. This step is trig- 
gered when the user submits a request form in the web 
UI. See Table 1. 


In order to figure out which tasks are necessary, 
the job creation algorithm relies on each module’s 
type; see Table 2. 


| Detail 
Source code that needs to be 
compiled. 


Website conten 
vul 


Content for our voice-driven 
service. 


Packages that need no compilation; 
vendor binaries, tools, and 
configuration files fall into this 
















Module Type 


platform 










| category. 


Table 2: Deployme module types. 


The rules are generally very simple at this stage. 
For example, platform modules need a BUILD step; 
the others do not. 


For the PUSH step, Deployme first finds out if 
the package 1s already on the destination server. If the 
package is already present, we skip the PUSH step, 
because we assume that packages never change. 


This optimization makes rollback extremely fast. 
No data needs to be copied to the server; instead, we 
progress immediately to the LINK step that simply 
reactivates the old package. 


If a PUSH 1s necessary, the job creation algo- 
rithm determines what route the package should take 
to the destination. In some cases, a single PUSH step 
is all that is required to copy the package from the 
Deployme master package repository to the remote 
location. However, if the final destination is at a 


Lifecycle Deployme 
Stage Task 
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remote facility, Deployme will instead add PUSH] 
and PUSH2 steps. 


The PUSH1 step sends the package to a machine 
at the remote facility which is designated as a gate- 
keeper. This machine serves as a cache of the master 
repository. The PUSH2 step asks the gatekeeper 
machine to copy the package from the cache to 
another machine at the same facility over the high- 
speed local network. 


Since most of our machines are in remote loca- 
tions connected by low-bandwidth connections, this 
logic saves hours of file transfer time. 


The Database 


Using a database on the backend makes it incred- 
ibly easy to provide a rich web UI. Also, a database 
allows operators to create highly refined reports using 
SQL queries. Deployme requires that the backend sup- 
port SQL and the Perl Database Interface (DBI). We 
selected the freely available MySQL RDBMS 
[MySQL2000] as its cost and speed were attractive. 
Also, the current version of Deployme does not 
require any advanced database support such as trans- 
actions. 


The database schema has tables that represent the 
visible items in the UI: modules, tags, and servers. 
There are additional tables, like “‘servergroups’’, that 
aggregate rows for a prettier display. Deployme can 
skip tasks that have already been done by previous 
jobs. The “‘state” table tells us which packages are 
active on which servers, and the “‘history”’ table holds 
jobs in all stages of execution. 


One problem with using a database to record 
machine state is that the database can become unsyn- 
chronized with the real world. For instance, if a given 
server 1s replaced or if an operator performs a manual 
package upgrade, the database will become stale. This 
issue was not much of a problem for content pushes, 
but as we extended Deployme to include platform and 
tool pushes to hundreds of servers, the inconsistencies 
began to pile up. 

We attack this problem through sanity checks 
and a reconciliation script. The sanity checks are per- 
formed both during job creation and execution. For 


Detail 


[Greatepackage PULL Export package from source control. 
Create package et 


BUILD | Build executable or content from sources or templates 
PUSH Push package directly to destination host. 


Distribute package 
PUSH2 

Install and activate package LINK 

Remove obsolete packages | CLEAN | 


PUSH1 | Push package to intermediate host. 
| Push package from intermediate host to destination host. 





Table 1: Deployme tasks. 
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example, if you attempt to push a SPARC package to 
an Intel machine, we stop the job before it starts. 


The reconciliation script scans the packages on 
remote machines and compiles a list of active versus 
Inactive packages. This list is compared against the 
database. The check script then updates the database 
to match the actual state of the server. 


The many problems we encountered with 
database consistency sparked a debate about whether 
the database should contain any state information at 
all. Another approach would be to make all job execu- 
tion decisions based on the actual state of the server 
instead of a cached version. However, we decided 
against local state in this version of Deployme due to 
the time it takes to inventory a single server and our 
desire to preserve the database’s excellent reporting 
capability. We will revisit this issue in version 2. 


Job Execution 


Most of Deployme’s code is in the job execution 
system. 

After job creation is complete, all tasks are filed 
in the database’s history table. The web UI then kicks 
off a background process to execute the new job. 

The job execution system is modular. We split 


Deployme into a set of Perl packages designed to han- 
dle different cases for each task. 


module name 
module_modtype 
module_contact 
module_comment 


state_status 


servergroups 


svrarp id 
syrgrp_name 


svrgrp_class 
svrgrp_push_svr_id 





state_tag “name 
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The first tasks are generally PULL tasks. These 
tasks use CVS to retrieve the appropriate tag for the 
given module. Once the tag is pulled into a package, 
the state table 1s updated. 


BUILD tasks are currently only performed for 
platform builds; the build task executes “make” and 
verifies build completion. The build module finds a 
build server in the server table and then uses ssh [Y]lo- 
nen1996] to remotely start and monitor the job. All 
build servers mount the master package repository via 
NFS and write the build results to the package, thus 
completing package formation. 


PUSH tasks are performed using either the rsync 
utility [Tridgell2000] over ssh, or the rsync utility with 
a listening rsync daemon. Configuration information 
in the database indicates which transport to use. Other 
transports can be added easily in the future. 


For PUSH2 tasks, we open an ssh connection to 
the intermediate server, and then initiate an rsync over 
ssh to the final server. 


We choose the appropriate code module for the 
LINK task based on the module type that is being 
pushed. 


Linking ‘““www”’ and “‘vui”’ modules types 1s rel- 
atively simple. Package activation often boils down to 
maintaining a few symlinks that point to the current 
packages. For our voice site, we also update the list of 
available keywords appropriately. 


his_date_complete 
his_svr_id 
his_tag_name 
his_release_status 
his_user 
his_logmsg 
his_parent_job 








Figure 2: Database schema. 
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The “platform” and “tool”? module types have 
the most complicated LINK step. We adopt the policy 
that any software that is running should be restarted 
when linked. For this reason, the LINK step needs to 
figure out which processes should be shut down 
before the link and restarted afterward. In some cases 
we also need to wait for processes to come back up. 


As producers and developers are free to do their 
own releases, the extent of our release engineering 
responsibilities boils down to when to run CLEAN 
jobs. Cleanup has its own job creation logic that deter- 
mines which packages are eligible for cleanup. For 
each module we specify how many old packages to 
keep on local disks and the minimum age of a package 
before it is eligible for deletion. Keeping some old 
packages allows us to do fast rollbacks since the pack- 
age is already at the destination — just a quick re-link 
is required. Using a minimum age is an attempt to 
make sure we have a good package to roll back to, and 
not just a series of bad packages that were released in 
quick succession. The cleanup logic examines the con- 
tent of each server and creates a list of packages to 
delete with the additional constraint that no active 
packages can be removed. A cleanup job is created 
that 1s executed similarly to a deployment job. 


The Problem With Restart 


The decision to add process restarts to Deployme 
proved to be much more difficult than we first real- 
ized. 


We first tried to create some general rules about 
what kinds of processes to restart on a machine, and 
only included certain critical processes in the rules. 
This group of special cases was small and we wrote 
them into the code base. As Deployme expanded, 
however, it quickly began to encompass many more 
programs than we had planned. 


This solution became a small dependency tree. 
This violates one of our Non-Goals discussed above. 
Because the dependency tree was restricted to plat- 
form LINK tasks, and because it remains rather small, 
we felt this transgression was minor. It turned out to 
be a continuing headache as we used Deployme for 
more modules and more servers. 


From a conceptual viewpoint, “‘restart’’ does not 
fit into the package lifecycle. We rolled it into the 
“Link and activation of package’? step, which 
increases the possibilities for failure in the LINK task. 
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There are significant ordering issues involved 
with restarting processes. While each task only relates 
to a given server/tag pair, the LINK step must aggre- 
gate all tasks for a given server together and then 
determine in what order to shut down the dependent 
processes. Order is determined using a small table in 
the database (not shown in the schema) that contains 
ranked module-process dependencies. 


Here is an example that shows how this can 
become complicated. Assume we have a job pushing 
new telephony firmware along with new monitoring 
software. The firmware requires that the telephony 
driver and the telephony server be shut down, where 
the telephony driver has higher priority. The monitor- 
ing software requires that the monitoring process be 
shut down. For maximum efficiency, the aggregator 
creates a job that will shut down first the monitoring 
software, then the telephony server, and then the tele- 
phony firmware. All links are done, and then the soft- 
ware is restarted in reverse order. We also add a spe- 
clal-case delay for the telephony firmware since it 
requires a half-minute to reconfigure after the load is 
complete. 


The ordering problem shows that it is not enough 
to just implement dependencies between modules and 
processes. We need to explicitly represent dependen- 
cles between processes and other processes. 


We were correct to not spend too much effort 
tackling the dependency problem early on. Our imple- 
mentation would have likely been incorrect. Having 
completed the first implementation of Deployme, we 
now understand what kinds of dependencies should be 
tracked. 


Parallel Remote Execution 


Another area of complexity is the method for 
triggering jobs on remote servers. 


The remote part of the platform link stage is too 
complicated to use individual ssh statements, so 
instead we created the “linkworker”’. The linkworker 
is a small Perl script with no dependencies that can be 
used to perform link, unlink, start, and stop operations 
on a remote host. 


Even if a link only takes a minute to perform, 
doing links serially across many servers takes a very 
long time. Serial processing severely inhibits 
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Figure 3: Deployme status. 


192 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Oppenheim & McCormick 


scalability. To help meet our scalability goal, we cre- 
ated a “parallel mommy”’ routine that parallelizes all 
links. 


When a linkworker is started, the mommy passes 
a “‘flag file’ argument. As the linkworker progresses 
through the jobs, it records what it has done in a flag 
file, and dumps logging output to a log file. Once the 
linkworker terminates, it prints “SUCCESS” or 
‘““FAILURE” in the flag file. 


The mommy routine starts all linkworkers in the 
background and then polls the servers to check on the 
flag file status. We poll instead of keeping our session 
open in case a transient network outage drops our net- 
work connection. As each server reports in, the 
mommy updates the job status and merges the remote 
log into the primary log. 

The linkworker reports to Deployme which pro- 
cesses were actually shut down. After the link, 
Deployme only restarts those processes that were pre- 
viously running. 


Error Handling 


A standard in Deployme 1s that user error should 
not result in a failed job. Instead, Deployme attempts 
to catch all user errors in the job creation step. This 
means that most of our users do not need to under- 
stand how Deployme works. These users just look for 
green boxes on the status page. 


Errors are handled using Perl’s structured excep- 
tion handling. This system catches all expected and 
unexpected errors and gracefully prints an error mes- 
sage to the web page or log file before aborting. 


All job execution actions are logged in text files 
that are stored for later review. The history table in the 
database tracks all jobs and stores the exception mes- 
sage if a job fails. Email is used for immediate notifi- 
cation of success or failure. 


Automatic Rollback On Failure 


Originally, part of our error handling strategy 
was to make individual tasks transactional. If we 
detected any problem with a task, we would set the 
state of the machine back to what it was before the 
task started. We found that automatic rollback on fail- 
ure worked better in theory than in practice. 


This policy only applied to platform and tool 
LINK tasks. Content LINK tasks rarely fail, and do 
not require any servers to be restarted. Failed PULL 
and BUILD tasks usually result from bad tags, and do 
not affect remote servers in any case. Failed PUSH 
tasks also do not affect remote server state because the 
PUSH logic copies the package to a temporary direc- 
tory and renames it to the actual package name once 
the copy is complete. This prevents half-completed 
packages from appearing on remote hosts. 


Initially, automatic LINK rollback seemed easy. 
The task 1s already separated into link, unlink, start, 
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and stop commands. Also, the parallel mommy rou- 
tine (discussed above) knows what tasks were com- 
pleted before the task failed. The obvious rollback 1s 
to undo whatever was done and call it quits. If the roll- 
back fails, exit immediately. If we are executing tasks 
in parallel, wait until they all report in and then initiate 
rollback for each one that failed, serially. 


The reasons for automatic rollback on failure 
were straightforward. We did not want to suffer down- 
time if a job failed, so we needed the machine restored 
to a functional state. Also, we felt that LINK rollback 
was necessary to prevent database inconsistency. We 
were afraid that individual failures over a period of 
time could make the database progressively more 
unreliable. 


We implemented the automatic rollback system, 
and it turned out to be such an operational failure that 
we tore the code out after several weeks. 


The problem was that all of the reasons cited 
above were wrong. 


Automatic rollback on failure became a major 
inconvenience for the network operations team. 
Instead of a job failing immediately so they could go 
in and fix it, they would be forced to wait while pro- 
cesses were restarted. The serial nature of the rollback 
made this maddening when multiple servers suffered 
similar failures. 


Besides the delay, it turned out that operations 
did not want the machine restored to its original state. 
Due to our network architecture, it is not a problem to 
have a single machine out of service. After a success- 
ful rollback, the first thing that operations did was shut 
down the server that rollback had restarted. 


Automatic rollback did not make the database 
more consistent. For many failed tasks, the database 
was inconsistent before the job began. Rolling back 
just went from one inconsistent state to the original 
inconsistent state. 


We learned from this experiment. In our case, a 
fail-fast system results in shorter outages than a sys- 
tem that tries to fix problems on its own. 


The Minimal Downtime Paradox 


We found that some of our goals were not 
exactly what our Operations team wanted. All of the 
link scripts meet our “Seamless Activation”’ goal to 
get the servers back up and running as soon as possi- 
ble. However, in regular maintenance scenarios that is 
not the usual Operations procedure. They often will 
take servers down for an extended time to perform 
other maintenance along with a software push. Also, 
some did not agree with the policy of “that which is 
on disk must be running.”’ 

As with rollback, we saw the paradox that 
restarting server processes automatically can result in 
longer outages, since operations will just have to stop 
and start the server anyway for their own maintenance. 
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In response, we added a feature to prevent server 
restarts from occurring during a LINK Job. 


War Stories 


Deployme does not have any UI for performing 
reporting, but its database is enormous. The tables 
contain sufficient information to reconstruct every 
software upgrade Tellme has ever made _ since 
Deployme was introduced. 


In several cases, we have used the database to 
ferret out information such as ““When did this package 
reach the production servers?”’ and “‘Which modules 
were upgraded since yesterday?” 


Deployme makes it so easy and fast to move to a 
previous package that our Operations team 1s rarely 
alerted when a bad package is released. Instead, the 
content producer quickly reverts to a good package 
and keeps the outage brief. 


In many ways, Deployme has reduced the num- 
ber of war stories we have to tell. The only major con- 
tent-related outages since Deployme was instituted 
resulted from individuals going around the Deployme 
process and editing files directly on production 
servers. (Old habits are hard to break.) There really is 
no reason to do such a thing in the Deployme world. 


Future Work 


Despite Deployme’s merits, the current imple- 
mentation 1s creaking loudly. The code base, approxi- 
mately 10,000 lines of code, has become difficult for 
us to maintain. 


The maintenance problem centers around the 
conditionals sprinkled throughout the code to check 
for a particular set of module types. For example, the 
“execute link” subroutine performs a large condi- 
tional to figure out which Perl module to call. Another 
example is that the web UI is littered with module 
type checks to determine what options to offer when 
creating a tag. 


Our solution for this is a concept called “‘ser- 
vices” which was introduced in [Finke1997], but our 
usage 1s slightly different. Instead of a module type, 
each module will be associated with a list of target ser- 
vices where that module can be deployed. Servers will 
provide services instead of having a single server type. 
The services table will contain a reference to the Perl 
object that will provide the service-specific code, iso- 
lated from the Deployme core. Additionally, we will 
add a START and STOP task to the list of deployment 
tasks so service restarts are distinct from the LINK 
task. 


One application of services 1s to allow Deployme 
to push to virtual web servers. Deployme only allows 
_web content to go to a single location on a remote 
server. With services, we can push to the “developer ”’ 
service on a web box separately from the “corpweb”’ 
service. 
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Our preliminary services plug-in implementation 
works very well. Because of this, we are moving all 
of the Deployme logic into objects descended from an 
abstract base class. For instance, CVS access will be 
abstracted under an SCS, or source control system, 
class. In this particular case, there has been some dis- 
cussion about moving to Perforce [Perforce2000] from 
CVS, and we want to be able to accommodate such a 
switch with little pain. 


All job execution steps and web UI are also 
being moved into subclasses. Once this is complete, 
the Deployme logic will be completely separated from 
the Tellme site logic. 


Another problem with Deployme is that it has no 
concept of a physical location shared between two 
machines. We investigated changing the database 
schema to know the difference between physical and 
logical locations, but the result was extremely convo- 
luted and hard to use. Instead, we plan to provide a 
mapping. table that designates a single server as the 
master for a shared mount point. At job creation, we 
will map any requests for deployment to a specific 
server/service pair to the master server if specified in 
the mapping. Since the job creation logic removes 
duplicate steps, we can be certain that we will deploy 
to a shared location only once. 


We plan to change the underlying database from 
MySQL to PostgreSQL [Lockhart2000]. PostgreSQL 
supports true referential integrity, which is necessary 
for the more complex Deployme2 schema. Also, Post- 
greSQL has transaction support, which makes error 
handling easier. 


Security is another area where much work 
remains. We want to build a granular system that lim- 
its what a user can do. Deciding how fine to slice 
access control is a tough question. To start, we will 
probably create ‘“‘roles” which have different capabili- 
ties. 


We need a way to quickly copy a package to 
multiple servers on a subnet. We have some prototype 
multicast file transfer programs written, but we have 
not yet decided how to incorporate them into 
Deployme. In addition to transferring bits via multi- 
cast, we could also blast commands to many hosts at 
once. 


Further enhancements include integrating 
Deployme into the system we use to initialize brand 
new machines. Also, we would like to add a “‘last- 
known-good” feature so that it is easy for operators to 
know what package is safe to roll back to. 


Status 


As of this writing, we are working on Deployme 
2. We intend to integrate many of the conceptual 
changes cited in Future Work. Most of the technology 
upgrades, such as multicast, are not a priority right 
now. This is because we are more interested in adding 
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the ability to add technology than the technology 
itself. 


Conclusions 


We originally wanted to find an off-the-shelf tool 
for accomplishing the goals stated above. However, 
none of the tools we reviewed provided an end-to-end 
solution for package lifecycle management. 


Deployme has greatly exceeded our expecta- 
tions. While it began as a simple content manager, it 
has expanded to become the upgrade center for our 
entire server environment. It currently tracks a few 
hundred servers, and we believe it can track thousands 
as we improve the underlying technology. 


The best thing Deployme has done for Tellme is 
something we never considered when writing up the 
goals. It has reduced our stress level. 


Stress is not something normally considered 
when writing tools. Usually we focus on scalability, 
speed, ease of use, and other immediately evident 
goals. When all of these immediate goals are met and 
the tool is firmly inserted into your process, that 1s 
when you see the secondary benefits. 


As Deployme began to encompass more teams 
within the company, our collective confidence grew 
and usage skyrocketed. Each day, Deployme processes 
40 to 60 jobs encompassing hundreds of individual 
tasks. 


Our development schedules are not held hostage 
to a release team. Instead, Deployme has engendered 
a release democracy, where even the newest employee 
1s empowered to take over a module and start sending 
new packages through QA and up to production. 


The best accomplishment of all is our most fre- 
quently asked question, ““That’s all I need to do?” 
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ABSTRACT 


Request-based distribution of software applications over a network 1s a difficult problem that 
many organizations face. While several programs address this issue, many lack features required 


for more sophisticated exports, and more complex 


solutions usually have a very limited scope. 


Managing these exports by hand is usually a time consuming and error-prone task. We were in 
such a situation when we developed the Automated Netdist program a year ago. 


Automated Netdist provides an automated mechanism for system administrators to request 
and receive software exports with an immediate turnaround. The system provides a simple user 
interface, secure authentication, and both user and machine based authentication. Each of these is 
configurable on a package-by-package basis for flexibility. 

Netdist is a modular service. The user interface, authentication and authorization are 
independent of the export protocol. We are currently distributing via NFS, but adding an additional 
protocol is as simple as writing a script to perform the export and plugging it into Netdist. 


Introduction 


At Indiana University, we have a large, extended, 
and heterogeneous Unix workstation presence admin- 
istered by many different departments and organiza- 
tions. The Unix Workstation Support Group (UWSG) 
is charged with supporting and advising the adminis- 
trators of these machines. This includes offering a 
variety of systems administration classes, negotiating 
and maintaining site licenses with various vendors, 
distributing a large variety of unix software, and main- 
taining a Unix Users Group. We also provide tradi- 
tional phone, e-mail, face-to-face, and extended con- 
tact support services. 


At the time Netdist was developed the UWSG 
consisted of six members, five full-time and one part- 
time, distributed across two campuses. This 1s a lot of 
ground for such a small group to cover, and we are 
always looking for ways to increase the efficiency of 
our services without sacrificing quality or our cus- 
tomers’ satisfaction. In early 1999, I was given the 
task of re-working one of our least efficient and least 
reliable services, the request-triggered distribution of 
software via NFS. 


Software Distribution 


The UWSG distributes a large and varied collec- 
tion of Unix software, to a widely varied audience. 
Software export laws, license agreements, pre-existing 
vendor-supplied distribution systems, varying security 
requirements, and machines with slow or non-existent 
network connection all work together to fragment our 
software distribution into the following components. 
These components are the environment out of which 
Netdist grew, and their attributes greatly influenced 
the development on Netdist. 

e Anonymous FTP 
e Secure HTTP Access 
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e Open NES exports 

e Request based NFS exports 

e Media Checkout 

e Vendor Specific Distribution Tools 


Anonymous FTP 


The majority of the software we distribute is via 
anonymous FTP. We maintain a large (15OGB+) site at 
ftp.uwsg.iu.edu. This FTP site is our preferred method 
of distributing files. This archive is available to the 
world and contains mirrors of most of the large Linux 
distributions, a CPAN mirror, a mirror of the Linux 
kemel archives, various freeware programs and utili- 
ties, and system patches. The benefits of this system 
are many: 

e It supports many concurrent users, and is lim- 
ited only by our current hardware and the uni- 
versity’s bandwidth. 

e It is available to anyone in the world with FTP 
access. 

e It is easy to use, and the ratio of support inci- 
dents to usage is orders of magnitude smaller 
than any other service we maintain. 

e It is efficient to administer, requiring little 
maintenance. 

e It is a very well known system, and administra- 
tion can be shared or passed on easily. 


Unfortunately, anonymous FTP is not a viable 
tool for all of our distribution needs. Following are 
several of it’s limitations that have forced us to dis- 
tribute packages using alternate methods. 

e Access control is very limited. We maintain a 
distinction between Indiana University (IU) 
clients and non-IU clients. Anything beyond 
this becomes much more difficult to administer 
as the target audience shrinks and the complex- 
ity of administration increases. 
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e Security is poor. FTP has a long history of 
security deficiencies, vulnerabilities, and 
exploits. We don’t make any sensitive data 
available via FTP, allow authenticated logins, 
or allow write access. 

We provide operating system media for system 
installations, and most operating systems aren’t 
able to be installed over an FTP connection. 
Several Linux distributions support this, but we 
must also provide Solaris, HP-UX, IRIX, and 
other Unix flavors that do not. 

FTP sites cannot be mounted as local file sys- 
tems. We offer several large software packages, 
and users might not have the free space to 
download a large software package, uncom- 
press it, and then install it. 


Secure HTTP Access 


For software that needs to be firmly restricted to 
IU affiliates we use an Apache web server running 
SSL and an in-house mod_perl module that allows us 
to securely authenticate users against the University 
wide kerberos database. This system is useful for dis- 
tributing smaller packages and we use it to distribute 
export-restricted security software, as well as IU-spe- 
cific license codes for several software packages that 
we have licensed. 


This service addresses the security failings of 
anonymous FTP, but doesn’t address the concerns of 
installing operating systems or dealing with large soft- 
ware packages. In addition, the server installation and 
configuration is much more complex than anonymous 
FTP, and the overhead on the server is greater. 


Open NFS Exports 


We maintain open NFS exports of several Linux 
distributions for network installs and updates. This is a 
convenient way for administrators to set up new 
machines over the campus network. As open NFS 
exports lack any real security, we do not export any 
other software this way. Indiana University does 
block incoming NFS requests from outside the school 
network, so the exposure 1s relatively small. 


Request Based NFS Exports 


At the time I began this project, we were export- 
ing Star Office (before it was purchased by Sun) and 
Sun’s Workshop Compiler Suite through NFS on a 
per-request basis. This allowed us to control who had 
access to the packages, while giving our customers the 
convenience of installing these packages as if they 
were mounted locally as a CD image. Our customers 
were happy with this service, but we were managing it 
by hand which was inefficient and error-prone. I was 
assigned the task of replacing this service, when we 
learned that several new software packages would be 
added to it. 


Media Checkout 


We maintain a large collection of OS and appli- 
cation CDs that are available for checkout by request. 
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This has been a very labor intensive service for us, as 
we must manage the physical storage of these CDs, 
make copies of popular requests, keep track of who 
has checked out what, and gently remind admins when 
they’ve kept a CD too long. A separate project to 
resolve the inefficiencies in this process is currently 
wrapping up, with wonderful results. 


This process is secure, and convenient for 
administrators to install from. It also provides those 
with only a modem or no network connection a means 
of acquiring software. It 1s, however, horribly ineffi- 
cient, generally requiring 15-30 minutes of work per 
request. We must respond to the request, obtain the 
physical media, schedule a pickup time, meet the cus- 
tomer, check their identification, register the checkout 
into our tracking system, and then collect the software. 


Vendor Specific Distribution Tools 


We use vendor-specific software distribution 
tools from both HP and SGI. Both of these tools use 
remote procedure calls for transferring software 
patches, updates, and new packages. These tools are 
convenient for administrators on campus to use, and 
required little administration. Both of these tools are 
proprietary software, and limited in the software they 
can serve and the clients that can access them. 


We use the IRIX Network Distribution server to 
distribute OS and Software updates as well as new 
packages. In order for a machine to be able to access 
the server, it must be listed in the servers .rhost file. 
This file tends to get rather large and outdated as time 
progresses, and must be pruned periodically. 


The HP Software Agent Suite allows us to make 
HP depot files available over the network to HP 
machines on campus. Depots are automatically avail- 
able to the whole campus, but implementing more 
granular permissions is a non-trivial task. The system 
uses a graphical interface to make patching and install 
software painless. 


Request Based NFS exports: The problem 


Under the old system, an administrator that 
wanted StarOffice or Sun WorkShop would send an e- 
mail message to our group account containing the 
software requested and the target machine’s hostname. 
Upon receiving the message, we would verify that al] 
of the relevant information was included, and if neces- 
sary request any information that has been left out. 
Then we would edit the NFS exports file by hand to 
include the client machine and a comment listing 
when the export expired. We then re-initialized the 
exports and notified the user that the export would be 
valid for three days. Once the export had expired, we 
removed the client hostname manually and then re-ini- 
tialized the exports to force the expiration. 


While this system worked, we knew it should be 
better. The ease of installation from an NFS export 
and the wide install base of NFS clients were nice, but 
there were problems with the system: 
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e Our response time to customer requests was 
inconsistent. Requests were only processed dur- 
ing business hours, and if we were teaching a 
class or were otherwise busy, it could take an 
hour or longer to process a request. 
The process was error prone, especially while 
editing the NFS export file by hand. Some 
problems we experienced were mis-typing 
hostnames, corrupting the NFS export file, and 
forgetting to re-initialize the exports. These 
were all easily and quickly resolved, but 
increased response times and introduced short 
service outages. 

Handling requests interrupted other tasks. 

While handling a request usually only took five 

to ten minutes, the interruptions could be diffi- 

cult to deal with. We place a high priority on 
response time, so even important tasks would 
be interrupted to handle a request. 

e The system was not scalable. The time required 
to process exports and the chance of error both 
increased linearly with the number of requests. 
With anonymous FTP adding an extra 100 
users, would not be noticeable, but the same 
increase in NFS requests would have required 
us to open another full time position. 

e Removing exports was problematic. It was easy 
to forget about an export after three days and 
not remove it from the exports file. There were 
also occasions where a hostname was added 
without comments, which made it difficult to 
determine if an export should be removed. 


Requirements in a Replacement 


After cataloging the strengths and weaknesses of 
each of our distribution systems, we constructed a list 
of features required in a replacement: 

e Automatically process user requests. 

e Automatically clear expired requests. 

e Only answer requests from IU machines 

e Must not require any proprietary software on 
the client machine. 

e Must be easy to use and to administer. 

e Logging must be complete. 

e Be able to authenticate the user, preferably 
through our existing Kerberos database 

e Any existing solution must be available free of 
charge. 


With such specific requirements, we had serious 
doubts about finding an existing software package that 
met, or even attempted to address, our needs. We had 
been prepared to develop the solution from the begin- 
ning, and already had several ideas about how to 
implement the service. Before proceeding with this 
implementation, we did perform a limited search for 
existing software that met our needs. 


We searched several common search portals, 
such as dejanews (now deja), yahoo, and altavista. 
Plodding through hundreds of results from various 
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search terms provided very few software distribution 
systems, and all of these were designed to face prob- 
lems different from ours. 


Failing to find an existing solution to our prob- 
lem, we began work on an in-house solution. We 
wanted to maintain the benefits of a file system based 
export, and since only a handful of machines on cam- 
pus had AFS or DFS, we decided to stay with NFS as 
the means of exporting the software. 


We decided to leverage our experience with the 
secure web server interface to our kerberos database to 
provide secure authentication, and a web _ based 
request form. This gave us the ease of use of NFS, the 
security of HTTPS through user authentication, and 
nearly ubiquitous installation of our clients, 1.e a web 
browser and NFS client. 


With export requests and authentication handled 
through HTTPS, and the software exports done via 
NFS, we needed something to tie the pieces together. 
We looked at several different models before narrow- 
ing our selection down to either a system of setuid 
root C programs, or a client/server interface, with the 
web server acting as the client to a custom designed 
server. We implemented rough versions of both of 
these, and did some performance and security testing. 
From the results we decided to go with the 
client/server model. 


A Brief Version History 


The initial version of Netdist was designed to 
alleviate the immediate problem with exporting 
StarOffice and Sun WorkShop. We were eager to 
release the system, and as such were unable to imple- 
ment several of the more ambitious planned features. 
The initial release in June, 1999, included the follow- 
ing features: 

e Automated user authentication. 
e Automated export initialization and expiration. 
e Ability to include or exclude sub-networks of 
machines by IP number on a per-export basis. 
e Variable export durations for different pack- 
ages. 
e Core API for implementing common methods. 
e 24/7 availability, with immediate response. 
e A client/server model with PGP based encryp- 
tion. 
The initial release went well, with positive user feed- 
back. As it became certain that we would be adding 
ApplixWare and Island Office to our available so ft- 
ware, we returned to Netdist development. In Novem- 
ber 1999, we released a much improved Netdist with 
ApplixWare and Island Office added to the list of 
available software. The improved features included: 
e Expanded API, and more encapsulated system 
overall. 
e Ability to limit exports to specific users. 
e System for managing export documentation. 
e Ability to control exports from multiple 
machines from a single web interface. 
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e Ability to accept plug-ins for exports of proto- 
cols other than NFS. 
e More efficient coding, faster response time on 
requests. 
Since this revision, Netdist has been a nearly ideal ser- 
vice to administer. It has required intervention less 
than a handful of times, and the logs show continued 
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Figure 1: Information flow between Netdist compo- 
nents. 
Architecture 


The Netdist program is a system of four modular 
components that can each be upgraded and extended 
as needed with minimal effect on the other compo- 
nents. The components consist of an API, client, 
server (with configuration file), and export control 
scripts. 


Briefly, the client is a web form that authenti- 
cates and authorizes each export request. If the request 
is validated, the client passes an encrypted request to 
the server. The server processes this request, and calls 
the appropriate export control script, depending on the 
export protocol of the software requested. The results 
of these exports are then returned to the client for pre- 
sentation to the user. The API ties all of these sections 
together by providing common procedures and data 
structures. Upon startup, the server reads a 
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configuration file that defines the valid export types, 
export restrictions, and other settings described below, 
and shares this information with the client. See Figure 
1 for an illustration of how these different components 
work together to manage software exports. 


The API 


The API is written in object-oriented Perl, and is 
available to the client, server, and access control 
scripts. The API is composed of six modules and con- 
tains any code that should be accessed from more than 
One component in order to increase efficiency and to 
ease communication between the different compo- 
nents. 


The API provides procedures for accessing and 
manipulating the data structure created by reading in 
the configuration file. This allows each component of 
netdist to manipulate this data structure. The API also 
manages all logging in order to guarantee comprehen- 
sive and compatible log entries from events logged in 
different components. API functions are also available 
for creating unique temporary files and performing 
date comparisons. 


The Client 


The client, or user interface to Netdist is a collec- 
tion of mod perl scripts available on an Apache 
Server with SSL. We use a custom Apache module to 
authenticate users against our kerberos database, but 
other Apache authentication schemes could be used. 
Once the user is authenticated, they are presented with 
a list of the exports they are authorized to request. 
This list comes from the data structure which the 
server creates from the configuration file, and is later 
passed to the client upon request. 


The user then enters the hostname he wishes to 
export the software to, and selects the desired software 
packages. The client performs taint checking on the 
data before processing it to look for illegal hostnames, 
and attempts to compromise the web server. The client 
then makes sure the the hostname entered is an IU 
machine, performs a reverse DNS lookup to make sure 
the hostname and IP number match, and confirms that 
the selected exports are valid for that host. 


If these conditions are met, the client opens a 
connection to the server and submits an encrypted 
request for the approved exports. The client waits for a 
success or failure response from the server and returns 
a message for any error that may have occurred. Oth- 
erwise, the client returns success, with links to instruc- 
tions for accessing the exports, and an expiration date. 


The Server 


The server is a long-running Perl daemon that 
must run as a user privileged enough to manage the 
exports. In our case, NFS is used and the daemon runs 
as root. At startup, the daemon processes the configu- 
ration file, and then listens for connections on a UNIX 
or TCP port, depending on how it is configured. The 
server uses a PGP authentication scheme to ensure 
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that it only processes requests from the client. There 
are two valid requests a client can send to the server. 
This first is for a copy of the configuration data struc- 
ture, and the second is to export a software package. 
Configuration requests are responded to, and export 
requests are mapped to the appropriate export control 
scripts, and results of the export are returned. 


The Export Control Scripts 


Each type of software export requires its own set 
of export control scripts. Each set consists of one 
script the server calls when requesting a new export, 
and a second script to clean up expired exports, which 
should be run by cron or another scheduling utility. In 
the case of NFS, the exports controlled by Netdist 
must be specially formatted to allow the control 
scripts to process them. Other exports can be listed 
normally after the final Netdist export. On a Solaris 
machine the formatted entry looks like this: 


4F@workSO begin Island exports 
#hostl.indiana.edu userl 2000/09/18 
d#Fhost2.indiana.edu userl 2000/09/18 
#Fhost3.indiana.edu user2 2000/09/19 
HHEwork50 end Island exports 
share -F nfs -o ro@hostl.indiana.edu: 


NOStZ aA ndtana edu?’ nostj3.2ndiana.edug 


filler.indiana.edu /is/netdist/ws50 


A comment entry ts added with each export containing 
the hostname, the username of the requester, and the 
date of the export. The comments not only allow the 
expiration scripts to determine when an export has 
expired, they also make the export file more informa- 
tive to human readers. 


Configuration 


We designed the Netdist program to be highly 
and easily configurable. From working with many 
Unix programs, we felt that a simple yet powerful 
plaintext configuration file was the best way to 
achieve this. The format of the configuration file, and 
the API calls for processing it, were the first sections 
to be implemented. 


We had an initial list of values to store in the 
configuration file, which grew as we developed the 
other sections. We attempted to extract every config- 
urable value to this file, but there are a few values that 
we missed, and will extract in the next version of Net- 
dist. The Netdist configuration file defines the avail- 
able exports and their properties expressed as key- 
value pairs. The valid keys are shown in the subsec- 
tions below. 


export 


This value is a short string that uniquely identi- 
fies the export within the system. It must be defined 
for each export, and is only displayed in the logs. 
Examples: (star5 1, is]60) 


text 


This value is a text string that identifies the 
export to a Netdist user. This should contain the full 
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title of the software package, the version number, and 
the platforms on which the package is valid. This 
parameter must be defined for each export. Example: 
( StarOffice 5.1 (Solaris|Linux)’) 


type 

This value defines the type of export the package 
is. Currently the only valid choice is nfs, but if other 
export protocols are added, this key will be used to 
determine which scripts to call to perform the export. 


host 


The host value sets the host on which the export 
is located. The host must have a configured and run- 
ning Netdist server. With this value a single web inter- 
face can manage exports from multiple servers. 


port 


This value defines the port on which the appro- 
priate Netdist server will be listening. This allows Net- 
dist to be easily introduced into environments without 
worrying about any given port being free. 


duration 


The duration value defines the number of days 
an export is valid before being expired. 


manual 


This value defines the name of the file to link to 
when providing instructions for acquiring and 
installing an export once it 1s approved. Instructions 
should be in HTML format. 


users 


The users value defines the list of users allowed 
to access this export. A value of ‘“*” will allow all 
users to export the software. Alternatively a list of 
usernames can be provided. Only users able to export 
the software will see it as an option. 


allow 


This value defines IP ranges to which the soft- 
ware can be exported. Each entry is expressed as a 
subnet mask, and multiple entries are allowed for each 
export. 


deny 


This value defines an IP range on which the 
export is not permitted. This can be used to remove 
part of a previously allowed network, including hosts 
and subnets. The syntax is the same as the allow 
parameter, and again multiple entries are allowed. 


Configuration: An Example 


The first export defined in the configuration file 
is a default. Any keys that are unassigned in subse- 
quent export definitions will use the default value. 
Below is an abridged configuration file, and an expla- 
nation of the entries. 


The First line declares the beginning of the 
default entry. The next few entries declare that the 
default protocol will be NFS, and indicate the the host 
and port on which to look for the server on. Next the 
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file specifies that by default all users will be able to 
access an export, and then gives the valid IP ranges for 
exporting software. 


Default 

type=nfs 

host=netdist.domain.edu 

port=86e 

users=* 

Ao wwii 

aso wells nl a. 

export=softl 

text='Software Application \ 
i MES Oda icy. (aac) 

manual=softappl.html 

al Lowa ll 113 

ae Vow= iad le 

deny isa? 

export=soft2 

text= “Software Application 21 (IRI 

manual=softapp2.html 

deny=1i1. 11 2%:1 


The export key defines the beginning of a new 
export. The value given becomes an internal identifier 
for the export and must be unique. The text and man- 
ual definitions function as described above. The two 
allow statements are combined, and they then replace 
(not expand) the default allow value for this export. 
The deny is then masked over the new allow value. 


The second export defined is similar to the first, 
but different in a few key ways. First, the export value 
is a different string. This is essential as each export 
string must be unique within a netdist system. Also, 
since no allow key is defined, the default allow values 
are used and the deny value given is masked over 
them. 


Below is a partial dump of the data structure cre- 
ated when the configuration file is read in. This struc- 
ture is then passed on to the web client so that it can 
only lists the exports a user has access to and to make 
security checks against the requests before passing 
them on to the server. This dump was created with the 
Data::Dumper Perl module. 


SVAR1 = bless( [{ 
“charg? => of 
deny: ==> |i); 
"AUISerie* “Sap er 
type “=7 mes. 
DOR s=7 22 POs 
‘toot =>“ Starorraee 5.2 < 
’ (Solaris: Linux); 
“COC. => stars 2 t ml. 
WOU Gato =p, 
“NOS spo Server.indtana-educ.. 
"allow’ => [ 
PeeWee led anor e 
ei clare vee. 
aula a 
] 
} 


‘default’ => { 
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Peay => Ing 
Miser Ss mp ot. 
"cViDe ae. ts. 
BDO Ries = pee Oh 
‘hext =~ Mderault 
“dirativon =?" 3:, 
‘HOSt? =? “Server windiana.edu 
alow’. J=y+ ib 
po ae Dee 
ees PL gel. a 
PLS ge! Sas oy 
] 
Ns 
"Wwork50" =>: -{ 
‘deny =~ ih. 
MiGere! => =) 


‘Ey pe” <7 nEs 
PDO. =) ee Ox 


‘text.. => “Sun WorkShop 5.0 (Solaris), 


"doc’ => ‘workshop50.html’, 
“duration: <=2 72 
“HOSt. => ) Server indiana. edu, 
‘allow’ => [ 
Pe il ordi Qos! 
] 
J 


hes “CONTASUrat Lon: Oa 


Security 


Security was a priority throughout the develop- 
ment of Netdist. We designed the architecture around 
several basic security tenets. 

e Users need to be authenticated before accessing 
the service. 

e Need to securely process NFS requests (which 
require root access), through our web server 
which runs under an_ unprivileged www 
account. 

e Logging must be thorough and structured to 
make potential security problems highly visi- 
ble. 


Authentication 


We have experience using apache with open-ssl 
along with with an internal mod perl module to 
authenticate web connections against IU’s kerberos 
database. These components had worked well for us in 
the past,allowing us to authenticate any IU user, and 
ensure that their usemame and password were safely 
transmitted to the server. Once authentication has been 
performed, Netdist works with the user through an 
abstract interface. Other apache authentication tools 
can be used, and the resulting data can be structured to 
work with the abstract database. 


Request Processing 


We required a method of responding to requests 
that was secure against local users as well as network 
based attacks. We investigated several options, includ- 
ing setuid programs, before deciding to use a 
client/server interface. They allowed us to securely 
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make the transition from web server to root permis- 
sions, and seemed more expandable than other solu- 
tions. 


After the web form processes the request, it 
makes a socket connection to a server running as root, 
and transmits the usemame, machine name, and export 
request. The server then processes and logs the request 
and sends the results back to the web form for display 
to the user. To ensure a robust service, Netdist sup- 
ports both TCP and UNIX sockets for its client/server 
communications. 


We must ensure that the server only replies to 
requests from authorized clients, that it encrypts any 
data sent over the network, and that the server handles 
malformed connections properly. Using UNIX sockets 
for communication simplifies this somewhat, as you 
only worry about attacks from the local machine, and 
can use file permissions to limit access to the socket. 
For this reason, UNIX sockets are preferred for Net- 
dist, unless you need the ability to control exports on a 
foreign machine, in which case TCP sockets are 
required. 


We use PGP to secure communication over the 
sockets. We generate a public/private key pair, assign 
the private key to the root account, and place the pub- 
lic key in the web server account’s (www) keyring. 
The public/private nomenclature of PGP encryption 
isn’t truly appropriate, as the public key 1s also pri- 
vate, and known only to the www account. No other 
copies of these keys exist, and both files can only be 
read by the owner. Neither of these accounts uses PGP 
for any other purposes, and the key pair is not used in 
any other way. 


In effect, this configuration allows us to encrypt 
and sign the message with a single key pair. Since the 
www client is the only entity with access to the public 
key, any message encrypted with it has been signed by 
the client. Initially, we used a second public/private 
key pair to sign the encrypted message in the tradi- 
tional PGP form. However, we eventually realized that 
the second key pair didn’t increase security. Our www 
client’s private key was no more secure than the 
server’s public key, and only served to complicate the 
communication protocol. 


This encryption/signing model relies on keeping 
the two keys a secret. This is a significant concem, as 
one of the keys is owned by the www account. Web 
server accounts are common targets of crackers as 
they often provide easy access to a machine. Not only 
must we worry about an attacker exploiting an inse- 
cure cgi script or server setting, it is also trivial for any 
user able to host cgi scripts on the web server to write 
a script that will print the keyfile. To reduce this risk, 
only members of the UWSG have accounts on the 
machine that hosts netdist, and only the www and root 
account can host web pages on the secure web server. 
Very few web scripts are run on this secure server, and 
all are closely examined, and perform strict taint 
checking on any entered data. 
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Each message sent by the client is encrypted 
with the public key. The server forks off a child pro- 
cess for each incoming message. It then goes through 
a series of steps to confirm the validity of the message. 
If any of these steps fail, the message and test failed 
are logged and the child process drops the connection 
and terminates before proceeding to the next step. 
These steps are completed in the following order: 

1. The server confirms that the message is PGP 
encoded, and does not contain any additional 
text or data. 

2. The server decodes the PGP message to ensure 
it was sent fiom the client. 

3. The server performs taint checking on the com- 
mand to ensure that it is a valid netdist com- 
mand with no special characters, shell escapes, 
etc. 


Once these tests are passed, the data is passed as 
a parameter to the appropriate export control script, 
and logged. At no point is code received by the server 
executed by perl or the shell. These precautions are in 
place to limit the impact of any successful attacks 
against the encryption/signing scheme. First and fore- 
most, arbitrary programs can not be run as root, and 
the aggressive logging should allow us to identify any 
successful attacks and close the service until the prob- 
lem is resolved. 

Logging 

Netdist maintains three different log files for 
readability. The first file logs any errors that occur 
during the export request process. This is the log 
where attacks on the system will be documented, as 
well as any problems that might be affecting the sys- 
tem. As such, it 1s important to closely monitor this 
log. 

The second log file logs the time, username, sup- 
plied hostname, and requested exports of every 
request. This log gives a comprehensive listing of the 
requested exports, and can be used 1n conjunction with 
the error log to look for suspicious access patterns. 


The final log records successful export requests, 
including the username, hostname, and timestamp of 
the request. This file can be be used to generate many 
different statistics on usage. It can be coupled with 
machine and user databases, to answer questions such 
as "How many requests were made by graduate stu- 
dents last semester?", or "How many requests for Star 
Office have been made from the Chemistry Depart- 
ment?". 


Distributed Netdist 


The Netdist program supports a distributed 
model, where a single web client can control software 
exports from multiple machines via multiple servers. 
This is useful for controlling multiple exports that 
can’t be hosted on the same machine. This model 
requires a single master machine which contains the 
netdist configuration file, an instance of the Netdist 
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server and the Netdist API. Additional machines that 
will export software are slaves and must contain an 
instance of the Netdist server, the Netdist API, and the 
export control scripts for each type of export is offers. 
The web client can be installed on any machine, and 
acquires its configuration from the master server. See 
Figure 2 for an example of how multiple machines can 
work together in a distributed Netdist enmronment. 


Target Machines 


Target Machines 


ie ellen 
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Exports 






Export 

Control 

Netdist L Script 
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Export a 
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Master Netdist Server 







=xport Requests 
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| Apache | 
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Figure 2: A sample Distributed Netdist implementa- 
tion. 





Portability 


Currently, Netdist could be ported to other sites 
and platforms with a modest amount of work. It 
requires a host with Perl 5 and some modules from 
CPAN, PGP, cron (or some other scheduling routine), 
and an instance of Apache with at least mod_perl and 
preferably a module for secure transactions. The NFS 
export control scripts have been written for Solaris, 
but could be modified to work with the syntax of other 
Unix flavors. 


The Perl modules and several scripts do have 
host or port specific information coded into the script, 
in a few places. Each of these instances is docu- 
mented, and easy to update. In the next version of Net- 
dist these values will be extracted to a configuration 
script to increase portability. 
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The Future of Netdist 


The development of Netdist has slowed, as I no 
longer work for the UWSG, and have less time to 
work on it. However, the following items have been 
completed/are being worked on: 

e Scripts to allow Netdist to control https access 
via .htaccess files 

e Scripts to allow Netdist to control DFS and 
AFS exports via group membership 

e Migrating from PGP to SSL for encrypting 
communication between the client and server. 
This is dependent on SSL support for perl 
through Net::SSLeay becoming more robust, or 
me re-writing the client/server in C. 

e Modifying the user interface to support many 
more available packages via a tiered menu sys- 
tem. 

The flexibility of Netdist is important to us, as pre- 
ferred protocols and programs can change quickly. 
Netdist allows us to maintain a single, familiar point 
of contact for customers, despite changes in the way 
data is moved. For example there has been a push at 
IU for DF S and AFS in the past few years, and Netdist 
is ready to work with controlling access to software 
available on these systems. 


Availability 


Netdist is still pre-alpha in that we haven’t done 
much work to ease installation, and we would like to 
incorporate some of the features mentioned above to 
increase it usefulness. As such, Netdist is not yet 
widely available. If you are interested in Netdist, 
please contact the author for the software location, and 
help with setting it up. 
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ABSTRACT 


Good UNIX system administration practice includes among its many tasks the proper 
configuration of system files, installation and maintenance of third party software, and 
maintenance of system security, including regular updates of operating system (OS) patches. For a 
small number of systems running only one or two OSes, keeping up with these tasks isn’t too 
difficult. However, as the number of systems and OSes increase (and the number of staff remains 
constant), these chores can quickly become overwhelming. 


This paper describes our planning, development, and deployment of a system that provides 
automated software distribution, patch installation, and OS configuration through the integration 
of GNU cfengine [Bur95], MySQL [MySQL00], and a few custom written Perl scripts. It is meant 
to be less of a tool description and more of a discussion about the various aspects of designing a 
multi-platform software and patch distribution system, and the benefits of integrating those 
systems into a configuration management system such as cfengine. Designing and developing our 


system has been a time-consuming endeavor, but it has proven to be well worth the effort. 


Background 


Our network was out of control. In our depart- 
ment, we have over }00 UNIX systems running more 
than half a dozen UNIX based OSes (more than a 
dozen when counting different OS versions). The vast 
majority of the systems share a similar role, but they 
are all configured slightly differently to suit their par- 
ticular users’ needs. With only two systems adminis- 
trators, it was extremely difficult to handle the day to 
day maintenance for each of these systems. We fell so 
far behind that the majority of our days were spent 
merely fighting fires, Because of this, we had very lit- 
tle time to respond to requests for software updates or 
new software installations and even less time to make 
sure that all of our systems were running at the current 
OS patch level. This left us with two large problems: 

e Most of our systems had very old copies of 
software. Since there was so little time to spend 
upgrading software, we would only upgrade or 
put new software on a machine when a user 
would specifically ask for it. It’s not hard to 
imagine what this led to; we ended up with a 
number of different versions of the same soft- 
ware all installed slightly differently across our 
systems, This made upgrading software a much 
more difficult task than it had to be, so unless 
there was a specific need for it, software 
wouldn’t get upgraded at all. 

Most of our systems were unpatched against 
known security problems. The chore of manu- 
ally applying patches to over 100 systems one 
by one every time a new patch report comes out 
is enough to give most systems administrators 
nightmares. Because of the great effort 
involved, most of our computers only had 
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whatever patches were current at the time of 
OS install and whatever patches were applied 
to the systems after mass break-in attempts. 


It was clear to us that we needed some way to 
manage the distribution and installation of software 
and patches for our systems. We felt that eliminating 
these two problems more than justified however much 
time would be spent developing a new management 
system. 


Requirements for a New System 


Over the period of several days, we brainstormed 
about what our ideal system should include and came 
up with the following rough list: 

e Software distribution and management 

e Patch distribution 

e Centrally controlled configuration 

e Ease of use 

e Low cost 

e Security 

e Flexible host configuration 

e Centrally controlled “pull” of software and 
patch distribution 

e Portability 

e Autonomous operation 


Software Distribution and Management 


Our system needed to be able to handle the dis- 
tribution, installation, upgrade, and removal of most or 
all of the software not supplied by the OS vendor that 
was needed on our computers. Additionally, it would 
be useful if our system could track information about 
installed software packages, such as a list of all files 
included in a software package, the original source of 
the software, and how it was compiled. 


207 


Use of Cfengine for Automated, Multi-Platform Software and Patch Distribution 


Patch Distribution 


Our system needed to be able to handle the dis- 
tribution and installation of all vendor supplied OS 
patches. It should also be able to handle any post- 
installation steps required by the patch, such as restart- 
ing a patched daemon. For maximum security, it 
would need to make sure that all of our computers 
were as up to date with their specific OS’s patch list as 
possible. 


Centrally Controlled Configuration 


As much of each machine’s configuration as pos- 
sible (e.g. automounter configuration, printer setup, 
firewall configuration, syslog configuration, etc.) 
should be initiated and tracked by a central system. 
Likewise, the list of all software and patches which 
should be installed on each machine should be man- 
aged by a central system. The exact configuration state 
of each machine in the system should be reproducible 
in the event of a hardware failure or any other event 
that would require a fresh OS install. 


Ease of Use 


Our system had be easy to use once placed into 
production. A system that was complex and difficult 
to use wouldn’t be much of an improvement over 
manually maintaining our systems and would likely go 
unused. The system should also be easy to maintain 
and require a minimal amount of work to keep opera- 
tional. The less work it required, the more time we 
could devote to other projects. The more time we 
could devote to other projects, the more productive we 
could make our users. 


Low Cost 


Our system must be inexpensive. We have a min- 
imal operating budget and can’t afford multi-host 
licensing fees and yearly software maintenance costs. 
This unfortunately ruled out most commercial soft- 
ware options. 


Security 


Security was a primary motivation for our under- 
taking this project. Our new system must allow us to 
achieve a higher level of security across our network 
than we previously had. This necessitated that our sys- 
tem be able to maintain a current OS patch level on all 
of our users’ computers. It should be able to quickly 
upgrade software across all systems whenever security 
bugs were found in third-party software. Any central 
servers and processes used by our system should also 
be as secure as possible. 


Flexible Host Configuration 


We can classify our machines as belonging to 
one or more specific groups (1.e., NFS servers, labora- 
tory workstations, members of specific research 
groups, etc.). Depending on what group(s) a machine 
belongs to, our management system should be able to 
install specific software packages and make specific 
operating system configuration changes. For example, 
machines grouped as laboratory workstations may 


208 


Ressman & Valdés 


need additional data analysis packages installed on 
them, while only machines belonging to a specific 
research group will need a print queue defined for a 
printer which belongs to that research group. 


We also needed to be able to manage software 
and OS configuration on a host by host basis. Our sys- 
tem should also be able to customize a machine’s soft- 
ware and OS configuration beyond the configuration it 
receives by virtue of its group classification. 


Finally, it should also be possible to customize a 
machine’s configuration files based on any particular 
third-party software packages or vendor supplied 
patches that are installed. For instance, for machines 
with the Apache web server installed, we need a way 
to manage the server daemon’s configuration and log 
files. 


Centrally Controlled “Pull” of Software and Patch 
Distribution 


The software and patch distribution, while con- 
figured centrally, should work on a system where the 
patches and software are “pulled” from a central 
server, rather than “pushed” by the server. By having 
“‘smart clients’”’ and a “dumb server,” we hoped to 
minimize the amount of damage that could be done to 
our system by the temporary loss or compromise of 
any one host. In our ideal situation, any or all of the 
functions of the server could be moved to different 
computers with a minimal impact to the system. 


Portability 


Our system must be portable. Our needs require 
it to run across multiple OSes, including SunOS 4.x, 
SunOS 5.x, IRIX, AIX, Digital/Tru64 UNIX, 
OpenBSD and Linux (including multiple architec- 
tures, such as Intel and Alpha). Fortunately, we do not 
need to support any non-UNIX OSes at this time. 


Autonomous Operation 


Most importantly, our system needed to do all of 


the above with as little intervention from us as possi- 
ble. 


Home-grown or Public Domain Software? 


At this point, the only thing we knew for sure 
about our new system was that it would have to have 
three major parts to it: software distribution, patch dis- 
tribution, and configuration management. The ques- 
tion we next asked ourselves was how much existing 
software could we use, and how much would we have 
to write ourselves? Our problems certainly weren’t 
unique, and we didn’t want to reinvent the wheel. 
Likewise, we aren't software developers and didn’t 
have the time to write a complete system from scratch, 
sO we wanted to make use of as much pre-existing, 
maintained software as possible. 


Software Distribution 


We spent a few days looking around on the web 
for software distribution systems but were only able to 
find a handful of references. When we took a more 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Ressman & Valdés 


careful look at the few systems we were able to find, 
we noticed that most of them relied on package direc- 
tory trees being pushed out from a server (rdist style), 
copied over from an NFS server, mounted from an 
NFS server with symlinks set up in the client’s local 
directory structure, or some combination of the three 
(Xhier [Sel91], Depot [Col92], Depot-Lite [Rou94], 
GNU Stow [Gli96], opt_depot [Abb97], SEPP 
[Oet98]). 


We had already ruled out a server pushed distri- 
bution system, so the only other choices for existing 
systems were distribution through AFS or NFS from a 
master server. AFS was not an option we could con- 
sider because it was not available for all of our target 
platforms nor was it freely available for most of our 
platforms. While NFS may have been an easy choice 
for the authors of these other distribution systems 
because of a pre-existing NFS architecture, it was not 
an easy choice for us. We had made very little use of 
NFS in our department for software sharing or distri- 
bution. 


Historically, we had always avoided running 
software mounted from an NFS server in favor of each 
machine having local copies of all of its software. In 
doing so, we minimized the amount of damage the 
loss of any one computer or the loss of our network 
could do to any other computer. We’re employed to 
keep our users as productive as possible by allowing 
them to think about astronomy and astrophysics and 
not about whether the NFS server that is holding the 
software they need to use is available. Inevitably, 
computers and routers will crash or have to be brought 
down for maintenance; we didn’t want to use a distri- 
bution system that would cripple all of our users’ com- 
puters through the unavailability of a single server. 
Having our software distributed from or mounted on 
an NFS server would increase our dependence to a 
specific server more than we felt comfortable with. 


Additionally, given our requirement to support 
multiple operating systems and architectures, using an 
NFS-based distribution system would have required 
that we either maintain a separate NFS server for each 
OS/architecture combination or maintain a compli- 
cated, multiple-architecture directory structure on a 
single NFS server. Lastly, we have software which 
must be installed locally on each system, such as soft- 
ware containing kemel modules and security related 
software (e.g. tcp-wrappers, ssh, etc.); if we used an 
NFS-based system, we would still need something 
else to manage the locally installed software. 


After ruling out server push and AFS/NFS, we 
were unable to find any free software distribution sys- 
tems which we could put to use in our department. It 
was clear that our best option was to write our own 
software distribution system. 


Software Packaging 


We needed our system to handle the installation, 
removal, and upgrade of 100 or more different third- 
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party software packages. Since we had ruled out a net- 
work filesystem based distribution method, we imme- 
diately realized that we would have to use some sort 
of packaging system to get our software from our dis- 
tribution server(s) to our client computers. 


We came up with the following three options: 

e Use the native package format for each OS. 

e Create a home-built package format using shell 
scripts and tar. 

e Use an existing, multi-platform package for- 
mat. 


Of these three, we immediately ruled out using 
each OS’s native package format for software distribu- 
tion. While it (arguably) might have allowed for the 
most trouble-free integration of the software distribu- 
tion system with the client computers and their OSes, 
it also would have been the most work to set up and 
maintain. Since each OS has its own unique packaging 
format, every piece of software we wanted to dis- 
tribute would have to be packaged up in as many as 
eight completely different ways. The last thing we 
needed was to make packaging more complicated than 
it had to be. Besides, we had had enough experience 
with some of the package formats to know that a few 
of them were complicated at best and downright 
obtuse at worst. 


A home-built package format using tar and shell 
scripts would certainly be the easiest to put together, 
since every OS we were using had tar and a Boume- 
compatible shell. Given that, no additional software 
would have to be added on the computer in order to 
install, upgrade, or delete packages. All we would 
need to do is: 

e Write shell scripts to install, remove, and query 
our available packages, 

e Compile each software package once for every 
OS for which we wished to have the package 
available, 

e Tar up the software into our package format, 

e Finally, put the tarball up on our distribution 
server(s) to be pulled in by our clients. 


It did have one drawback; we’d still need to get 
the package from the server to the client. Since we’d 
ruled out NFS, the only reasonable ways we could 
think of to get the package to the client over the net- 
work were HTTP or FIP. That alone wouldn’t be 
enough to stop us from using a home-grown package 
manager. We could install a program like GNU weet 
or NcFTPget when we loaded the shell scripts on the 
client. It would be bare-bones and wouldn’t have a lot 
of fancy bells and whistles, but it wouldn’t be a bad 
solu tion. 


When looking around for a Few Good multi-plat- 
form package formats, we came across a couple lesser 
known ones that looked like they might be able to 
work, but the clear leader in that field is the Red Hat 
Package Manager (RPM) [RPMOO]. Aside from 
already having been ported to every operating system 
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on which we would need to run it, RPM has a very 
active development group, a very broad user base, and 
has been thoroughly tested. 


RPM also extends beyond just a packaging for- 
mat into a development environment that can control 
the entire packaging process from the compilation of 
software from source code on up to the installation of 
the packages. It also has the added benefit (or hin- 
drance, depending on how you look at it) of extensive 
package dependency awareness, and has a built in FTP 
client so that it can retrieve packages from a remote 
FTP server and install them in one step. 


Building RPM packages would require more in1- 
tial work in that we would have to write a package 
specification (spec) file for each package. The spec 
file contains all the information that is needed to com- 
pile, install, uninstall, and upgrade the package for 
every OS for which the package would be available. 
However, we didn’t think this was necessarily a bad 
thing. It would enable us to document from where we 
got the package’s source code (not always an easy 
thing to remember or find). It would also very clearly 
show us every step we would need to take in order to 
compile and install the software (again, not always an 
easy thing to remember). Additionally, once we had 
written the spec file, updating packages to newer ver- 
sions of the software would usually be very easy. 
Often, all that’s required to build a new RPM package 
when updated versions of software are released is a 
one-line change to the package’s spec file followed by 
an “rpm -b -a package.spec.”” The RPM manual 
[Bai99] contains an excellent description of the entire 
process of building an RPM. 


The only problem we saw with using RPM 
would be the initial installation of RPM itself. We 
decided that if we were going to use RPM, we would 
build an RPM package in each OS’s native package 
format, install that package on each system, and let 
RPM install all the additional software packages. 


While the tar and shell script option would cer- 
tainly have sufficed, we felt that the added functional- 
ity of RPM (especially the FTP client) was worth the 
effort of leaming how to write the RPM spec files and 
building RPM packages for each OS. The tar and shell 
script option also would have required that we rein- 
vent much of the functionality already in RPM. There- 
fore, we felt our best option was to write our software 
distribution system around the Red Hat Package Man- 
ager with packages distributed from an FTP server. 


Patch Distribution 


However disappointing the lack of information 
on available software distribution systems was, the 
lack of information on patch distribution systems was 
doubly so. We were unable to find any information on 
multiplatform patch distribution and installation sys- 
tems. In fact, the only useful information we were able 
to find about any kind of automatic patch download 
and installation software was a reference to a program 
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named PatchReport in an article in the October 1997 
issue of ;login: [Sin97]. PatchReport was a Solaris- 
only Perl script that would compare the current patch 
state of the machine it was being run on against Sun’s 
patch cross-reference file. PatchReport would down- 
load any patches that were missing and install them. 
Since a large fraction of our computers were running 
Solaris, this was a pretty good start, but since it was a 
Solaris-only utility, 1t was clearly not going to be our 
multi-platform patch distribution and installation util- 
ity. 

We were unable to find any such utilities for 
IRIX, Red Hat Linux, AIX, or any other OSes we 
would have to support. If we wanted a true multi-plat- 
form patch distribution and installation system, it 
appeared we would have to write one ourselves. 


Since patches are supplied by the OS vendor or 
development team, there was no way to use a common 
patch format. Our only choice would be to write a sys- 
tem that was intelligent enough to recognize what OS 
it was running on and download and apply the right 
patches. Since we were already going to use an FTP 
server to distribute our software, we decided to use an 
FTP server to distribute the OS patches as well. 


Configuration Management 


When we started the search for configuration 
management software, we immediately found hun- 
dreds of references to a program named cfengine. 
After a cursory glance at the online documentation we 
found on the cfengine web page, we decided that this 
software package was definitely worth a closer look. 
We downloaded the cfengine reference manual 
[Bur99] and went home for the weekend to read it. 
The more we read, the more we realized that cfengine 
was a perfect choice for our configuration manage- 
ment system. 


Cfengine would allow us to control every aspect 
of a machine’s configuration that we thought would be 
necessary and then some! A complete description of 
cfengine is beyond of the scope of this paper, but 
briefly, cfengine will let you manage: 

e Copying of files, both locally and remotely 
e Editing of files 
e Creation, removal, and maintenance of sym- 
bolic links 
e Filesystem access control lists (ACL) 
e File and directory permissions and deletions 
e Filesystem tidying 
e External command execution 
e System and user processes 
among many other things. 


All of cfengine’s actions can be conditionally 
applied based on whether or not certain ‘“‘classes”’ 
(cfengine’s name for “‘groups’’) are defined. Cfengine 
provides predefined classes based on the OS a given 
system is running, the hostname of the system, the day 
of the week, etc., and also allows you to define your 
own classes. Cfengine can also define classes 
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dynamically at runtime, so that classes can be defined, 
for example, only if a certain action was carried out. 
Given cfengine’s actions and ability to group actions 
based on class, it was clear that we could use cfengine 
to manage system configuration which applies glob- 
ally to all of our computers, to any individual com- 
puter, or to any predefined or dynamically defined 
group of computers, as we required. 


As an added benefit, cfengine has a framework 
for allowing users to write their own modules in any 
programming or scripting language and merge them 
into a stock cfengine run. Through this, you can add 
nearly any functionality to cfengine without having to 
actually modify cfengine itself. Modules can use and 
define classes as well, giving them the same control 
and flexibility as cfengine’s builtin actions. Because of 
this capability, we decided that we would implement 
our software and patch distribution system as cfengine 
modules so that we could use cfengine’s class mecha- 
nism to manage configuration based on installed soft- 
ware and patches. 


Summing It Up 


We now knew that our solution would include a 
home-written software distribution system built upon 
RPM software packages, a home-written patch distri- 
bution system, and cfengine for configuration manage- 
ment. 


Drawing Up a Rough Design 


Having decided on the features we would want 
and the software we would use and create, it was time 
to figure out what we would need to write into our 
software so that it would meet our requirements. 


Class Definitions 


As we’ve mentioned, all of our computers can be 
classified as being part of one or more larger groups 
that share a common functionality. There are personal 
workstations, laboratory workstations, data servers, 
computational servers, etc. The systems can be further 
divided into smaller groups; there are groups of 
machines that are used for a specific research project, 
share a common user base, need the same software, 
etc. Depending on which group a machine belongs to, 
it will have different software installed and its config- 
uration files will be different. 


To handle this varied configuration, we would 
simply make use of cfengine’s class mechanism. For 
this, we needed to define classes which reflected the 
grouping of our systems, and then create cfengine 
input files which would associate our systems with our 
classes and run whatever actions are necessary in 
order to configure the systems as needed. 


Central Configuration and Control 


We needed our system to centrally manage all 
system configuration. Having all configuration infor- 
mation resident on a central server makes it easy to 
change configurations for multiple systems with a 
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single edit, minimizing mistakes and eliminating 
inconsistencies between systems. Cfengine naturally 
lends itself to central configuration. In the most com- 
mon method of cfengine use, master copies of 
cfengine’s input files (the files that tell cfengine what 
commands to perform on the computer) reside on the 
cfengine server where they are pulled in by the client 
systems through cfengine’s internal network file copy 
protocol. 


We also needed our system to centrally manage 
all of the available software and patch information for 
all of the OSes we use with easy expandability for any 
future OSes we might need to support. Every single 
system needs to be able to talk to the central system 
and be told exactly what software packages should be 
installed and what patches should be applied. 


We came up with two different possible ways of 
distributing this information: 

e Have each client pull in a group of text files 
that describe the available software, each 
machine’s software configuration, and each 
OS’s patch configuration. 

e Have each client query a relational database to 
obtain its software profile and patch informa- 
tion. 


A text file option would be the easiest to set up. 
However, when we did the math to figure out how 
much data would be passed around in those files, it 
became clear to us that keeping track of over 100 soft- 
ware programs on over 100 machines would require 
more than 10,000 different entries in our text file — for 
the software subscription information alone. 


Having two people maintain this text file was a 
recipe for disaster. No matter how careful we were, 
we were bound to make syntax mistakes and typo- 
graphical errors. The more entries we had in this file, 
the harder it would be to track down any mistakes. 
While we could probably keep it under control with 
some difficulty, 1t was very clear that this option 
would not scale as the number of hosts and software 
packages increased. Maintaining a separate file for 
each computer with its own software information 
would have been just as bad. 


Having this information stored in a database was 
clearly the better choice. With properly written front- 
end scripts to maintain the database information, it 
would be very easy to keep the database relatively 
error-free, and make the inevitable errors very easy to 
find and correct. In addition, most databases are 
designed with scalability in mind and could hold far 
more detailed information for far more hosts than we 
would ever need to support. 


Hence, we decided to store all the software and 
patch information in a central database. Of the free 
database packages available, we. chose MySQL 
because we had had some familiarity with it in the 
past and because we felt that the development commu- 
nity was a little larger than the other two databases we 
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considered, mSQL and PostgreSQL. Any of the three 
would have been a suitable choice for our project. 


Portability 


RPM and cfengine had already been reported to 
run on every OS on which we would need to run them. 
To assure portability, all we needed to do was to make 
sure that our cfengine modules were written in a lan- 
guage that was portable across all of our systems and 
had the ability to communicate with a MySQL server. 


We made a list of all the scripting and program- 
ming languages we could think of with MySQL sup- 
port and came up with the following: Perl, Python, 
Tcl, C, C++ and Java. 


Of those, we immediately ruled out three 
choices. We decided not to write our modules in 
Python out of personal preference. We ruled out Tcl 
because the MySQL support was not as strong as we 
would have liked. We also ruled out Java because a 
Java virtual machine was not available for every oper- 
ating system we would need to run it on. 


With Python, Tcl, and Java gone, we were left to 
chose either Perl, C, or C++ which prompted an inter- 
esting question: were there any specific benefits or 
hindrances that made scripting languages a better 
choice than compiled languages (or vice versa)? 


While personal preference was bound to affect 
our decision, we tried to be as objective as possible. 
These modules would not be doing large amounts of 
heavy computation, so any speed increase given by 
choosing a compiled language was likely to be negli- 
gible. These modules would be distributed from our 
central server through cfengine, which would mean 
very little difference between distributing one Perl 
script to all OSes or distributing a binary for each OS 
built from the same source code. 


If we wrote our modules in Perl, we would need 
to make sure that every computer had a copy of Perl 
and a copy of the MySQL module for Perl. Whereas 
if we used C or C++, no additional software would 
need to be installed on the client machines; all we 
would need is a copy of the MySQL libraries and 
header files on one system running each OS in order 
to compile the C or C++ modules. 


However, if we used Perl, it would be highly 
unlikely that we would ever use any of its more com- 
plicated features that might cause our scripts to react 
differently depending on what operating system they 
were being run on. As a result, if our scripts compiled 
and ran on one OS, they would likely compile and run 
on all the others. We wouldn’t need to recompile our 
modules on eight different OSes every time we made a 
small change to either of our modules. Releasing new 
versions of the modules would also be easier because 
we wouldn’t have to compile and test the program on 
all of our OSes with every minor change. 


We eventually chose Perl because of the ease of 
development and maintenance it would give us over C 
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or C++. Every computer we would be responsible for 
maintaining would have a copy of Perl on it anyway, 
and it wasn’t much trouble to add the MySQL module 
to it. It’s not clear that this was the best choice, and in 
the future, we may rewrite our modules in C or C++ as 
an experiment. 


Autonomous Operation 


Building our system through the use of cfengine 
modules was the perfect solution for us. Because of 
the way cfengine handles class definitions through its 
modules, any of cfengine’s internal functions could be 
performed on our systems based on what patches or 
software packages we installed — all automatically! All 
that is required is that our modules define classes for 
the patches and packages that they install; we can then 
have cfengine run commands based on these class def- 
initions. 

The possibilities are almost endless. We can have 
cfengine restart daemons after installing OS patches 
that update them. Cfengine can change inetd.conf and 
send inetd a hangup signal if we install or uninstall a 
software package that is launched from inetd. Nearly 
any change that you would want to make to a com- 
puter due to the installation or uninstallation of a soft- 
ware package or OS patch can be performed on all of 
the applicable hosts by merely adding a couple of lines 
to cfengine’s input file. 


Below, we show an example of a cfengine input 
file which illustrates how classes can be used by 
cfengine to reconfigure inetd based on whether or not 
the IMAP server package is installed. 

editfiles: 
imapd:: 
( jete/anet/inetd cont 


AppendIfNoSuchLine "imap stream tcp 


nowait root /usr/sbin/tcp imapd" 
DefineClasses "inetd" 


} 


!imapd:: 

{ fete/inet/inetd.cont 
DeleteLinesStarting "imap" 
DefineClasses "inetd" 


} 
(Tater-“in “the: input file 
processes: 
rer: 
TIineta 2s." 
action=signal 
signal=hup 
In this example, we show the use of two builtin 
cfengine commands, editfiles: and processes:, together 
with a dynamically defined class called imapd::. The 
editfiles: command provides a number of actions for 
manipulating text files, while the processes: command 
provides actions for manipulating UNIX processes. 
Through a process detailed in the next section, our 
software module will dynamically define the imapd:: 
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class based on whether or not the IMAP server pack- 
age is installed; if the package is installed, the class 
will be defined, and if the package is not installed, the 
class will not be defined. 


Looking at the example above, you can see that 
if the imapd package is installed (i.e., the imapd:: class 
is defined), cfengine will check to make sure that the 
imap service is enabled in the inetd.conf file. If a line 
for imap doesn’t appear in inetd.conf , that means that 
the package has just been installed, at which point 
cfengine will append the line given in double quotes to 
Inetd.conf and then define the inetd:: class (this class 
will be defined if and only if cfengine edits the 
inetd.conf file as a result of this check). Later on in the 
cfengine run when the processes: command runs, 
cfengine will see that the inetd:: class is defined and 
send the inetd daemon a HUP signal as instructed by 
the input file. 


If the imapd package is not installed, then the 
imapd:: class will not be defined by our module, and so 
cfengine will run the actions in the !imapd:: stanza. 
These actions tell cfengine to remove the imap entry 
from inetd.conf if present and set the inetd:: class if the 
entry was removed. If the imap line 1s present, that 
means that the package has just been uninstalled. In 
this case, cfengine will remove the imap line from 
inetd.conf and define the inetd:: class, which will later, 
when the processes: command is run, cause cfengine 
to send a HUP signal to inetd. 


Similar actions can be taken with the installation 
or removal of any software package or OS patch. 


Summing It Up 


We now had a good idea of exactly how we 
wanted to implement our system. It would be driven 
by cfengine so that every machine could be configured 
centrally from the cfengine server. We would have a 
MySQL database holding up-to-date patch informa- 
tion for each of our OSes and software configuration 
profiles for each of our hosts. We would need to write 
two cfengine modules in Perl: one to make sure that 
the computer was up-to-date on patches, and one to 
make sure the computer had the proper third party 
software installed. 


Putting It Together (The Hard Part) 


Cfengine and RPM 


The first step in making this system materialize 
was to build a cfengine server and familiarize our- 
selves with cfengine in operation. For our cfengine 
server, security and economy were our top priorities, 
so we bought a Pentium-based system and installed 
OpenBSD on it. We disabled every service except for 
ssh and ftp, and loaded a highly restrictive ipfilter 
ruleset. We downloaded and installed cfengine and set 
up cfengine’s daemon to share input files with all of 
the systems on our subnets. We picked a handful of 
computers on which to install cfengine. These com- 
puters would contact our cfengine server every hour to 
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check for updated input files, download them if neces- 
sary, and run the commands in the input files. After 
two or three weeks of leisurely experimenting, we felt 
we were familiar enough with cfengine to begin 
designing a software and patch module system around 
il. 


Since a large fraction of the computers we’re 
responsible for are Sun SPARCs running Solaris, we 
started developing on our Solaris machines first. After 
making and installing a Solaris pkg package for RPM, 
we thought of all the third party software that we 
would want to install, downloaded the source code, 
and compiled them into RPM packages. This took 
about two months and was by far the most time con- 
suming step of the entire process. 


Software and Patch Database 


Once we made all of the RPMs, the next step 
was to design and create a database to store all of our 
software information and patch information. Towards 
this end, we had to get into the specifics of exactly 
what kind of information we wou Id want our database 
to hold. For software, we came up with the following 
list: 

e The name of every package available. 

e A listing of what OS each package was avail- 
able for. Some of our software would only need 
to be compiled for one or two of our operating 
systems. For example, our Red Hat Linux sys- 
tems would already have installed out of the 
box several packages which we would be build- 
ing for our other systems. We needed to be able 
to differentiate, for example, between the 
‘“screen”’ package that shipped with Red Hat 
and the one that we built from our own spec file 
for OSes that didn’t ship with a copy of screen. 
It would be very nice for our systems to be 
aware of which packages are installable and 
which ones not to bother with. 

The current, default version of each software 
package, applicable to every operating system. 
The ability to specify different current versions 
of the software packages for different operating 
systems. Some versions of a few of the soft- 
ware packages we’d be using simply wouldn’t 
be runnable on some of our operating systems. 
It'd be great to say, ‘Everybody run version 
1.24 of the ‘foo’ package, except for you AIX 
machines; I want you running version 1.22.” 

A list of every software package that should be 
installed on every machine. 

A field in each machine’s profile that would 
allow you to specify a specific version of a 
software package if you wanted a version other 
than the current. Some of our users are running 
software on their computers that require spe- 
cific versions of packages like Perl or Tel. It 
would be necessary to keep these packages 
from being replaced when we released new ver- 
sions and marked them as being current. 
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Figure 1: Database table holding software information. 


After we had come up with that list, we decided 
to use two database tables to hold this information. 


One table holds information for each software 
package we want to distribute (see Figure |). Each 
package has an entry in this table. One column in the 
table specifies the default version number of each 
package; this is the version of the package that 1s 
applicable to all OSes. Additional columns specify the 
version number of each package for each OS (Solaris 
Version, Linux Version, etc.). This version overrides 
the default version of a package for the given OS. If 
the value in the OS version column is NULL, then that 
means that the current version of the package for that 
OS is given by the value in the default version col- 
umn. Finally, there are columns which are used to flag 
the availability of each package for each OS (Solaris 
Build, Linux Build, etc.). For each package, if the 
value in these columns is non-NULL for a given OS, 
then that means that we have built an RPM of that 
package for that OS. This can be used to distinguish 
between RPMs which we have built and those that are 
included with the OS. 


For example, looking at the entries given in Fig- 
ure 1, we can see that an acroread RPM 1s available 
for both Solaris and Linux, and that the default version 
of the acroread package is 4.05-1 for all OSes. On the 
other hand, a tcl RPM is isn’t available for Linux (it 
wasn’t necessary to make one, since Linux already 
includes tcl) but is for Solaris. Finally, we see that the 
default version of the pam_opie package is 1.1-1, but 
that Solaris systems should use version 1.0-2 while 
Linux systems should use version 0.21-3. 


The second table holds information specifying 
which software packages should be installed on which 
machines (see Figure 2). For each entry in this table, 
one column specifies the hostname of the machine for 
which the entry applies, a second column specifies the 
name of the package that should be installed on that 
machine, and a third column specifies which version 
of the package should be installed on that machine. If 
the version is listed as ‘“‘current,”’ then that means that 
the current version of that package as given in the 
software table should be used. The collection of all 
entries for a machine in this table makes up that 
machine’s software profile. 


To illustrate, refer to the sample entries in Figure 
2. From here we can see that the machine called 
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‘““mypc”’ (which is running Linux) should have the 
current version of acroread installed. As we saw 
before in Figure 1, the current available version of 
acroread for Linux is 4.05-1, so that means mypc will 
have acroread version 4.05-1 installed. Similarly, we 
can see that this machine will have pam_opie version 
0.21-3 installed. Finally, we can see that machine 
‘““mysun” (which is running Solaris) will have tcl ver- 
sion 7.6-1 installed, even though the current version of 
tcl in the software table is 8.1.1-1. 


Package Package 
Name Version 


pam_opie current 
Tabet 


Figure 2: Database table holding host software pro- 
files. 





OS 


Patch No. ; 
Version 


107451-04 


solaris 


Figure 3: Database table holding patch information. 





The list we came up with for the patch database 
table was much simpler — the only thing we needed 
was a simple list of every patch for every version of 
every operating system we’d be maintaining. The 
table we used for the patch database is illustrated in 
Figure 3. 


Once the database was designed and created, we 
needed to populated it with our software and patch 
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information. Since we would be the ones making all of 
the new software packages and deciding what soft- 
ware gets installed on what hosts, it would be easy for 
us to keep the software database up to date. Every 
time we released a new package, or wanted to install a 
piece of software on a host, we would just update the 
software database tables, either directly through the 
MySQL command line client, or through a front end 
(web-based or otherwise). 


Keeping the patch database up to date would be a 
trickier affair. We would have no control over when 
new OS patches were released, and very few vendors 
or development teams would send us email whenever 
new patches were released. We would have to take the 
initiative of regularly going to each OS’s patch infor- 
mation center, downloading any new patches, putting 
them up on our ftp server, and updating the database 
so our clients could pull them in. 


To be clear, we didn’t think going out to every 
OS’s patch information site to check for new patches 
every moming would be a bad habit to get into, we 
just thought that we could find better things to do with 
the time that it would take to do that. 


We decided to automate that process and run 
scripts from our cfengine server to go out and do it for 
us. Since these scripts would only be run from our 
central server, we could take existing software (such 
as PatchReport) and modify it to download new 
patches to our FTP server and optionally update our 
database. For OSes that we could not find any such 
software for, it was easy enough to write quick Perl 
scripts to check that vendor’s WWW or FTP site, 
download new patches, and update the entries in our 
database. This way, we could check much more often 
than we could if we were doing it ourselves. Currently, 
we check every OS’s patch information site every four 
hours, around the clock. Whenever the script down- 
loads a patch, it emails us a notice containing a one- 
line description of each downloaded patch. 


We should probably mention one precaution, 
however. In general, it is probably not a good idea to 
install a patch on a system without first examining any 
documentation that comes with it and trying it out on a 
test system first. The documentation will mention any 
special steps which need to be taken before or after the 
patch 1s applied, and prudent testing will uncover any 
potential problems that may be caused by a malformed 
patch. 


In our current system, our patch download script 
automatically updates our database each time it down- 
loads a new patch from the vendor. This means that 
our client systems will pull in and install the down- 
loaded patches without our intervention. Given that 
we are only checking for “recommended” and “‘secu- 
rity” patches (the two patch types that seem to be well 
tested by the vendor before release), we aren’t overly 
concemed about any potential damage resulting from 
automatic updates. We may revisit the wisdom of that 
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decision. If we do, all that will be necessary will be a 
quick change to disable the portion of the download 
script which updates the database. We could then just 
manually update the database after we have inspected 
and tested a patch. 


Software and Patch Modules 


Once we had the cfengine server and the soft- 
ware and patch database infrastructure built, all we 
needed was the client software. 


From this point, it was relatively easy to write a 
cfengine module in Perl to grab a host’s software pro- 
file, download the RPMs and install them. Since RPM 
was chosen to be the common package format across 
all of our OSes, no changes had to be made to this 
script in order for it to run on the different OSes. 
When run through cfengine on a system, this module 
will: 

e query the MySQL database to generate a list of 
all software packages and their versions which 
should be installed on the system, 

e generate a list of all software packages and 
their versions which are currently mstalled on 
the system (by locally running “‘rpm -q -a’’), 

¢ compare the two lists and install, delete and/or 
upgrade any packages on the system as needed 
(by locally running rpm), 

e define a cfengine class for each software pack- 
age that remains installed on the system. 


After the module completes, cfengine continues 
its run and can act on any class definitions activated 
by the module. In this way, cfengine can make any 
configuration file edits, file copies, process signaling, 
etc. which are necessary as a result of any software 
changes, as illustrated with the imapd example in the 
previous section. 


The patch module was more tricky to write. 
Since all of our OSes handled patches in completely 
different ways, it was a challenge to make one script 
that would seamlessly patch as many as eight different 
OSes. It would be unavoidable to have certain parts of 
the script dedicated to only one OS, but we wanted to 
maximize the amount of code that would be common 
to all the OSes. 


As with the software module, the patch module 
operates by first querying the MySQL database to 
generate a list of patches that should be applied to the 
current OS. It then compares this list to a list of 
patches currently mstalled on the system. Finally, it 
downloads and installs or updates patches as needed, 
using the native patch mstallation command for the 
OS. The module defines a class for each patch that it 
installs for further use during the cfengine run. For 
example, 107451-xx is a patch for the cron daemon in 
Solaris 7 for SPARC. Whenever a version of this 
patch is installed by the patch module, it will de fine 
the class 107451 which we can then use in our 
cfengine input files to tell cfengine to restart the cron 
daemon. 
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Some patches may fail to install on some sys- 
tems. For example, a patch may be for an operating 
system component which isn’t installed on the system 
(e.g., UUCP), or it may be intended for a specific type 
of system and not others (e.g., 64-bit systems and not 
32-bit systems). While we could use the MySQL 
database to specify the applicability of a patch to a 
given system, we decided to only track the OS and the 
OS version for which a patch is applicable. As a 
result, our patch module may try to install inappropri- 
ate patches on a given system. 


Fortunately, at least on the OSes for which we 
have so far implemented our patch system, the native 
patch installation tools which our module uses are 
intelligent enough to determine on their own that a 
given patch isn’t applicable to the system and fail 
gracefully without damaging the installed OS. As a 
result, we are comfortable with our module trying to 
install inappropriate patches. Should the native patch 
installation tools not have the requisite intelligence, 
we will have to expand our patch database structure 
and module to accommodate. Fortunately, our design 
is flexible enough that this shouldn’t be too difficult. 


Given that some patches may legitimately fail to 
install, the patch module tracks failed patch installa- 
tions using a text file on each system. When building 
its list of patches to install during a cfengine run, it 
will remove from the list any patches listed in this file; 
that way, it won’t needlessly try to reinstall previously 
failed patches each time cfengine runs. 


As a final step, both cfengine modules email us a 
report of their actions so that we can be kept aware of 
what software and patches are being installed on our 
computers. The report details which software pack- 
ages and patches have been installed and/or removed, 
and includes any failures, errors or unusual output 
generated by the modules. 


Summing It Up 


We had now set up our cfengine server and our 
MySQL database, made RPMs of all our software, 
written the software and patch modules, and written 
the scripts to keep our patch information current. It 
was at last time to roll it all out! 


Making It Work (The Fun Part) 


Before we had started any work on this system, 
we agreed that every computer that was to be inte- 
grated into our management system would have to 
have its OS completely wiped clean and the current 
version of its OS installed. We wanted to minimize the 
number of versions of any given OS that were being 
used in our department, and since security was one of 
our primary motivations for undertaking this project, 
we also wanted to be assured that the OSes were clean 
and untrojaned. 


We would start by integrating all of our Solaris 
systems, then our Red Hat Linux systems, and then 
our IRIX systems. That would cover about 95% of all 
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of our computers, after which we would declare vic- 
tory and integrate the other OSes as time permitted. 


We picked two Solaris machines with particu- 
larly understanding users to use as guinea pigs, com- 
pleted their software profiles, backed up their home 
and data directories, and marched off to install Solaris 
de 


Once we had a clean installation of Solaris, we 
loaded on the RPM package (in Solaris pkg format), 
installed the Perl RPM with MySQL support, and the 
cfengine RPM. Sun’s JumpStart program made this 
process extremely easy to do, all completely auto- 
mated. We ran cfengine for the very first time, expect- 
ing the worst, and waited for the email report. 


Much to our surprise, it worked flawlessly. All of 
the configuration files were modified just the way we 
had laid out in cfengine’s input files, all of the patches 
we wanted applied were applied, and all of the soft- 
ware we put in the database configuration profiles was 
properly installed. After inspecting the systems for 
anything that was out of place, we declared the debut 
of our system a smashing success and made plans to 
upgrade the rest of our Solaris systems. At the time 
this paper is being written, all of our Solaris systems 
have been upgraded, and we’re about halfway through 
our Linux systems. Upgrading and integrating the 
Linux systems has proven to be equally painless. 


So far, our system has not caused any serious 
problems, and we’ve repelled numerous attacks on 
vulnerable services because our systems are never 
more than a few hours behind on vendor-supplied 
patches. 


Successes 


In this section, we figured 1t would be appropri- 
ate to include a few examples of where our system has 
clearly shown itself to be more useful to use than our 
previous hodge-podge method of systems administra- 
tion. 


Breakin Detection and Termination 


Cfengine gives you an excellent interface to 
interact with your system’s processes. You can search 
for specific processes, start them if they’re not run- 
ning, send them signals if they are, and email you with 
a report of what cfengine has found or done. 


Since the people who break into our systems 
almost exclusively use the compromised systems to 
run sniffers, IRC bots, or DoS tools, we decided to 
make up a list of suspicious process names to have 
cfengine look for and warn us about every time it ran. 
Besides the usual suspects (more than one running 
copy of inetd, anything with “sniff”, ‘“‘r00t”, 
“eggdrop’’, etc. in the process name, password crack- 
ers, etc.), we had cfengine watch for any process with 
‘* 7” in the process name. 


One afternoon, we got an email from cfengine on 
one of our computers that had noticed that the regular 
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user of that machine was running a program as 
‘‘/irc’’. It wasn’t uncommon to see our uSers using 
‘*/ to run programs, nor do we have objections to our 
users running IRC, but in this case, it was a bit 
unusual for this particular user to be running an irc 
process (good UNIX system administration practice 
also dictates that you know your users). 


Poking around the system, we discovered that 
the person running this program was not the regular 
user of the machine, but was someone who had evi- 
dently sniffed our user’s password from somewhere 
else and remotely logged into his system just minutes 
before cfengine had alerted us. This person was in the 
process of setting up an IRC bot and had not yet tried 
to get a root shell. 


At this point, we had to figure out exactly what 
to do to minimize the amount of damage that this per- 
son could cause with our user’s password. We were 
about 40 minutes from the next cfengine run, so we 
had 35 minutes to make changes to cfengine’s input 
files so that they would be pulled in by all our clients. 
The machine that this person had broken into 1s part of 
a group of computers in a specific professor’s research 
group. Every user who works with this professor has 
accounts on every one of his machines, and almost all 
of the users use the same password on all of the 
machines. That meant that the person who broke into 
the one machine we had noticed had a password that 
would likely let him break into about 10 other 
machines. It was very important to us that we block 
them from being able to get into all these other 
machines. It would only be a matter of time before 
they figured it out and began installing programs all 
over this professor’s research workstations. 


Through cfengine, we had set up syslog on all of 
our clients to log all messages sent to the AUTH facil- 
ity to a central log server. Based on the logs on the 
server, we could say with relative certainty that this 
person had only managed to break into the one sys- 
tem. We also had the JP address that he had come 
from. We decided that we would boot the cracker out 
of the compromised account, lock out our user from 
all of the machines on which he had accounts, and 
block all IP access to every machine in this 
researcher’s group from the entire class C from which 
the attacker had come. 


In our Solaris cfengine input file, we added a 
rule for every computer in that professor’s group to 
look in /etc/shadow for the regular expression 
‘“user:.*:” and replace it with “‘user:*LK*:” (we 
don’t use NJS or an equivalent, so each system as its 
own local passwd and shadow file). We also made a 
change in the global ipfilter ruleset to drop and log 
every packet coming from the class C from which the 
breakin had come. 

Just before the next cfengine run took place, we 


killed off the intruder’s shell. Two minutes later, the 
user had been locked out of all of the research group’s 
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machines until we could get in touch with him and get 
a new password. The sniffed password the cracker had 
obtained was now completely useless, and he had lost 
all access to our machines from the host that he was 
using as a breakin staging point. We watched the logs 
and systems intently for the next few days and saw 
absolutely no sign that this person was trying to get in 
again. 

All within the time frame of one hour, we had 
detected the breakin, determined the cause, terminated 
it, and prevented it from happening again. With the 
old way of doing things, it’s doubtful we would have 
ever caught this person unless he had launched a DoS 
and caught the attention of our campus-wide network 
security team. Score one for the good guys! 


Breakins Prevented Through Proper Patching 


Far less dramatic, but just as satisfying, was a 
recent experience with a new Linux bug. 


In the old way of doing things, we would be rela- 
tively free of breakins (as far as we knew) for periods 
of time, and then some script kiddie would find an 
exploit and decide to go to town on the university. 
We’ve had days that began with the campus network 
security team giving us a list of 20 machines on our 
local subnets that they’d seen compromised and had 
pulled off the network. We would have to drop every- 
thing that we were working on for two days to clean 
up these systems. These breakins would occasionally 
be for newly found vulnerabilities, but more often 
would be for vulnerabilities that had been around long 
enough to have had patches released. We'd get hit 
because we hadn’t had enough time to patch any of 
our computers against the vulnerability. 


Recently, we’ve seen several attacks against a 
bug in rpc.statd on campus. When they first started to 
show up, we braced ourselves for the worst, but after 
comparing the CERT advisory that addressed this vul- 
nerability (CERT Advisory CA-2000-17) against Red 
Hat’s errata page and the patches we had installed, we 
found that our system had already downloaded and 
installed an updated version of rpc.statd that was 
invulnerable to this specific attack. We had been 
safely patched for weeks. 


Conclusions 


So far, our system has proven to be well worth 
the effort it took to bring it to fruition. We rarely have 
to worry about system vulnerabilities because all of 
our systems automatically patch themselves as soon as 
new patches are released. We update software regu- 
larly now because we can easily compile it once, and 
install it on dozens of systems simply by editing one 
entry in one database table. We have regained control 
of our network. 


Availability 


All of the scripts that we’re using to run our sys- 
tem, including the patch module, software module, 


217 


Use of Cfengine for Automated, Multi-Platform Software and Patch Distribution 


and the CGI script we use to configure the software 
profiles for all of our hosts are treely available at: 
http://astro.uchicago.edu/davidr/cfengine-tools/ . 
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ABSTRACT 


Due to the demand of 24x7 coverage by present day data centers, the need for a proven 
disaster recovery plan is a must. To assist in providing disaster recovery for systems running Sun 
Microsystems’ Solaris 2.x (SunOS 5.x) operating system, a tool was developed which captures the 
image of the system to one or more volumes of optical media with the first volume being bootable. 
The optical media used by this tool is CD (Compact Disc) with hooks put in place for DVD 
(Digital Video Disc or Digital Versatile Disc). 


This tool was developed with the following objectives in mind: (1) no magnetic media; (2) 
bootable media; (3) minimal user interaction; (4) handle multiple volume sets; (5) handle 
environments that do not use a Network Information Service (NIS or NIS+); and, (6) handle 
environments that do not use Network File System (NFS). The power of this tool 1s made possible 
by utilizing two special features present in the Solaris operating system, namely the installboot 
utility and the JumpStart mechanism (as implemented on the Solaris 2.x Install CD). 


The complete process of capturing and restoring a Solaris system’s image to bootable optical 
media involves five phases: (1) pre-imaging preparation; (2) setup of target host; (3) capture image 
of target host; (4) bum image to media; and, (5) restore image to target host. Each of these phases 
is controlled by one master Boume shell script. The overall tool is implemented in 26 Boume shell 
scripts controlled by the 5 master scripts. This tool, known as the “CART” (Capture And 
Recovery Tool), was placed on a mobile cart and consisted of the following components: 
UltraSparc 10, internal CD-ROM drive, internal floppy drive, 256 MB memory, 2 network 
interface cards (NIC), keyboard, mouse, monitor, external SCSI 18 GB disk drive, external SCSI 
CD-RW, and extemal SCSI DLT. 


This tool provides the following functions: (1) bare-metal recovery; (2) capture a snapshot of 
a system to optical media; (3) cloning a system; and, (4) rollout multiple clones of a system via 
optical media. Using the principles discussed in this paper and creating additional Bourne shell 
scripts, the CART has been modified to provide the following additional outputs: (1) make copies 
of the Solaris Install CD; (2) make customized Solaris Install CD (essentially a customized 
JumpStart from CD); and, (3) a specialized bootable CD for disaster recovery that assists third 


party Backup and Recovery utilities (Legato Networker and Veritas NetBackup). 


Introduction 


With the increase of e-commerce on the Internet, 
there is an ever-increasing demand to have mission 
critical computer systems run 24 hours by 7 days for 
long periods between scheduled down times. When a 
mission critical system suffers from corrupted disk 
data, either software or hardware induced, the disk 
data will need to be restored. Software induced data 
corruption include application malfunctions that ren- 
der data inaccessible or a user accidentally entering a 
data destructive keystroke such as rm -r. Hardware 
induced data corruption include physical disk crashes. 
In the case of physically damaged disk drives, the disk 
drive of course will need to be replaced prior to the 
restoration of software and data. 


When the disk drive in question happens to be 
the boot drive, the situation becomes more involved. 
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One cannot simply run the backup application to 
restore the system because there is no longer a viable 
operating system running (nor present) on the system. 
Under such dire circumstances, rebuilding a boot disk 
can take what seems like an inordinate amount of 
time. The operating system needs to be installed, stan- 
dard packages (clusters) installed, customized pack- 
ages installed, patches applied, kere] tuning, and 
finally any special or custom configurations need to be 
set. To perform all of these tasks could take up to a 
couple hours. Even using Sun Microsystems’ auto- 
mated install utility, JumpStart, can take what might be 
considered too much time, because following the 
installation of the operating system and any cus- 
tomized packages, the Jumpstart process will sti}! need 
to initiate the installation of the desired patch sets. The 
application of a large set of patches can take up to a 
couple of hours. 
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Due to the lengthy time it sometimes takes to 
patch a system, Hewlett-Packard’s automated install 
utility for the HP-UX operating system, “Ignite-UX,” 
can prove to be more efficient time-wise to install a 
system. When Ignite-UX is finished, the system will 
be complete with operating system, applications, 
patches, kemel modifications, and any specialized 
configurations. Ignite-UX uses the concept of saving 
an entire system image of a benchmark host, known as 
a “‘golden image,” and then restores the image during 
the automated rollout. The Ignite-UX process essen- 
tially performs a bit-by-bit copy, and therefore, is 
much faster than the JumpStart process, which builds 
the target system from the ground up by installing the 
operating system, software packages, patches, and any 
additional applications and files desired. 


Taking the best of both the JumpStart and Ignite- 
UX worlds, a tool for providing disaster recovery, 
cloning, or snapshotting has been created specifically 
for Solaris systems. 


The Capture And Restore Tool — The “CART” 


During its evolution, this tool passed through a 
client site that needed to maintain the utmost in secu- 
rity. As such, the client site did not run network ser- 
vices to ease the administration nightmare of main- 
taining the numerous users, hosts, and system admin- 
istrative files (usually kept in the /etc directory). Thus, 
this client did not use Network Information Services, 
NIS, nor its more secure brother, NIS+. This client 
also did not use the Network File System, NFS, to 
share resources over the network for fear that such a 
service was not secure enough for the very confiden- 
tial information that was stored on the network. In 
order to accommodate this client, the tool was made to 
be self-contained and lived on a mobile cart so that the 
tool could be easily negotiated through the data center. 
The system hosting the tool will be referred to as the 
“control host” while the system to be imaged will be 
referred to as the “target host.’’ The two hosts will 
talk to each other via a directly connected network 
cable. 


This mobile tool physically consists of the fol- 
lowing components: UltraSparc 10, intemal CD-ROM 
drive, intemal floppy drive, 256 MB memory, 2 net- 
work interface cards (NIC), keyboard, mouse, moni- 
tor, extemal SCSI 18 GB disk drive, external SCSI 
CD-RW, and extemal SCSI DLT. Additionally, the 
system has a network cable attached to one of the 
NICs and an RS-232 serial cable attached to serial 
Port B. The RS-232 cable was a convenience added to 
handle headless hosts via the ‘tip’ utility. 


The main purpose of the tool discussed in this 
paper is to capture and restore system images. Further- 
more, the tool physically resides on a mobile cart. 
Because of these two reasons, the tool became known 
as, and will be referred to during the rest of this paper, 
as the “Capture And Restore Tool,”’ or simply the 
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CART. Figure 1 diagrammatically depicts the physical 
layout of the CART. 


Two Special Solaris Features 


The secret and power of the CART is brought 
about by two very special features found in the Solaris 
2.X operating system. These two features are specifi- 
cally the installboot utility and the JumpStart mecha- 
nism. The former facilitates the placing of a boot- 
block on storage media while the latter facilitates a 
way to customize the resultant boot process. These 
two features are further explored below. 


How to Make Solaris Storage Media Bootable 


One of the objectives of the CART was to restore 
a system via bootable optical media. The first question 
to ask is, ‘Is it possible to make bootable media under 
the Solaris 2.x operating system?’ The answer is a 
resounding YES! The solution is to use the install- 
boot(1M) utility. The installboot utility, uniquely found 
in the UNIX command sets distributed in Sun 
Microsystems’ Solaris 1 x (SunOS 4.x) and Solaris 2.x 
(SunOS 5.x) operating systems, provides the capabil- 
ity to install a bootblock on various types of storage 
media. Furthermore, the bootblock can be placed on 
any of the slices (partitions) on the target media. 


The second question to ask is, ““Can a bootblock 
be placed on optical media?” Again, the answer is a 
resounding YES! As a matter of fact, the bootblock 
can be placed on various types of bootable media, 
such as a floppy diskette, hard disk drive, CD-R, CD- 
RW, CD-ROM, DVD-R, DVD-RW, DVD+RW, DVD- 
ROM, and DVD-RAM. When the bootblock program 
resides in the boot area of a disk partition, the boot- 
block program will load the boot(1M) program, also 
known as ufsboot. The ufs boot objects are platform- 
dependent, and reside in the /usr/platform/‘uname 
-I‘llib/fs/ufs directory, where ‘uname -i' gets the correct 
platform-name. The full command syntax for install- 
boot is as follows: 


installboot bootblock raw-disk-device 


where: 

bootblock is the name of the bootblock code. 

raw-disk-device is the name of the disk device onto 
which the bootblock code is to be installed; it 
must be a character device which is readable 
and writable. Naming conventions for a SCSI 
or IPI drive are of the format: c?t?d?s?. Nam- 
ing conventions for an IDE drive are of the for- 
mat: c?d?s? . 

For a thorough explanation of the installboot utility, see 

the Man Pages on installboot(1M). In short, the install- 

boot utility was used to install the necessary boot- 

blocks so that the CART could handle the various Sun 

Microsystems platform architectures. More on how 

this was accomplished later. 


A Brief Discussion on JumpStart 


There are plenty of books on the subject of Jump- 
Start [1, 3, 6], and as such, the scope of this paper is 
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not to teach JumpStart. However, a brief discussion of 
JumpStart will help elucidate its role and importance as 
implemented in the CART. The JumpStart feature was 
introduced by Sun Microsystems and SunSoft in 1992 
and first appeared in the Solaris 2.x operating system 
[3]. The JumpStart feature is primarily used to install 
multiple client hosts over the network simultaneously 
with the intent to lessen this tedious and time-consum- 
ing task. The various hosts can have their own unique 
configuration if desired or groups of hosts can have 
like configurations. JumpStart is a utility to help facili- 
tate the ease of rolling out the Solaris operating system 
to new hosts (or existent hosts to be upgraded) in an 
environment and is especially useful for rolling out a 
large number of installs across an enterprise [2, 5]. 


Typically, JumpStart is used over a network to 
allow the automated software installation rollout of 
multiple hosts simultaneously. Once configured, the 
JumpStart mechanism is very much hands off. To facil- 
itate the JumpStart mechanism over a network, a net- 
worked JumpStart server is needed. But JumpStart 
servers are not the only place that one will find the 
implementation of JumpStart. Sun Microsystems has 
also incorporated the JumpStart mechanism on the 
bootable installation media (i.e., the Solaris 2.x Install 
CD) as the means for installing the Solaris operating 
system. 


The JumpStart mechanism has four main parts: 
(1) a BEGIN script which performs pre-processing 
actions to take place prior to the software installation; 
(2) a PROFILE which defines how Solaris is to be 
installed on a particular client; (3) a FINISH script 
which performs any post-processing actions following 
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the software installation; and, (4) a rules file (rules.ok) 
that dictates which BEGIN script, PROFILE, and FIN- 
ISH script to use. 


Because JumpStart offers the following three 
characteristics: (1) can be configured to have minimal 
user interaction; (2) can be placed on optical media; 
and, (3) can be invoked seamlessly following a boot, 
JumpStart was selected hands down as the method of 
choice for the CART. 


As will be described in the next section, the 
CART ended up customizing JumpStart in a com- 
pletely different way and was able to unleash the true 
power of JumpStart! 


The “CART” in a Nutshell 


To make use of the existent JumpStart technology 
as implemented on the Solaris Install CD, the CART 
performs the following steps: (1) partitions a hard disk 
drive (which will be referred to as the “‘image disk’’) 
to have slices that mimic (size and location) the 
Solaris Install CD; (2) creates filesystems on the 
slices; (3) copies select contents of the Solaris Install 
CD to the corresponding slices on the “image disk”’; 
(4) opens up the permissions on the directory and files 
where the JumpStart mechanism resides; (5) replaces 
the standard issue JumpStart BEGIN script, PROFILE, 
FINISH script, and RULES.OK file with its own 
highly customized versions; and, (6) installs the 
appropriate platform-specific bootblock that corre- 
sponds to the platform of the “target host.’’ See Figure 
1 for a quick overview of CART’s hardware configu- 
ration. 


The "CART" 
Hardware Configuration 





Figure 1: CART hardware configuration. 
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At this point, if the contents of the “‘image disk”’ 
is burned to optical media, that optical media will be 
bootable and will go through an automated install 
based on instructions in the customized JumpStart files, 
specifically the customized BEGIN script and the 
RULES.OK file. The customized BEGIN script ends 
with a system call to reboot the system. This reboot 
essentially causes the normal JumpStart process of 
loading packages, patches, etc. not to occur. In short, 
the CART makes use of the Solaris Install CD Jump- 
Start mechanism only as a means to provide the auto- 
matic running of a script following the boot from opti- 
cal media. It was decided not to reinvent the wheel but 
instead to modify an existent proven technology. The 
steps just described are what truly give the CART its 
power. 


This technique can be used for producing a wide 
range of useful applications, some of which will be 
discussed later in this paper. The final implementation 
of the CART did not use a PROFILE or a FINISH 
script. However, a customized FINISH script could be 
implemented to initiate backup software to restore 
variable data from the latest backup of the system. 


After the “image disk”’ has been prepared suc- 
cessfully, the next task of the CART 1s to capture the 
entire system image of the “‘target host.”” To accom- 
plish this task, a private network is setup between the 
“control host” and the “target host” with NFS 
mounts established between the ‘“‘target host” and the 
“image disk” (located on the “‘control host’’). Next, 
the “target host”’ is interrogated to determine its num- 
ber of disk drives. The partition table of each disk 
drive is logged capturing the slices used and the sizes 
of those slices. The CART then methodically runs 
through each disk drive and captures an image of each 
slice. As the image for each slice is captured, the 
CART logs the size of the slice image to a file. The 
size of each slice is captured for two reasons. First, to 
determine if an individual slice image will be able to 
fit on a single optical media volume. This is important 
in the case when saving to CD because a single vol- 
ume can only hold 650 MB. The second reason is to 
determine how many optical media volumes will be 
required to save the entire system image of the “target 
host.” 


Once the entire system image of the “target 
host” is captured, which typically will be comprised 
of several individual slice images, the entire system 
image must be burned to the optical media of choice. 
The log containing the size information of the individ- 
ual slice images is used to determine which slices get 
placed on which volume. The user of the tool will be 
informed how many volumes will be required to burn 
the entire system image and as each volume is burned 
the user is requested to remove the volume and insert 
the next volume. 


Once all of the optical media volumes have been 
successfully burned, the result will be a snapshot of 
the entire ‘‘target system” captured onto a set of 
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optical media, with the first volume being bootable 
and having the smarts to recreate the “‘target host.”’ 


The last step to complete this eventful saga is to 
perform the restore of the image from the bootable 
media, either to the original “‘target host” or to an 
identical host with as large or larger disk drives. The 
restore takes place at the Open Boot PROM (OBP) 
level by booting from the optical media. 


The Five Phases of the ““CART” 


The capture and restore of a Solaris system 
image using the CART involves five distinct phases. 
Phase 1, the pre-imaging preparation of “image 

disk” phase, prepares a buffer area (hard disk 

space) known as the “image disk.” The 

“image disk” will have appropriate slices (par- 

titions) and filesystems created for holding 

select directories and files of the Solaris Install 

CD, as shown in Table 2. These select directo- 

ries and files contain the JumpStart mechanism 

and the bootblocks for the various Sun 

Microsystems architectures. In addition, the 

“Image disk”’ will have a separate slice to hold 

log files during the capture phase and another 

slice to receive the compressed filesystem 
images from the target host. 

Phase 2, the setup of target host phase, establishes 
a private network connection and NFS mounts 
between the ‘“‘target host’ and the “control 
host” that has access to the “‘image disk.”’ 

Phase 3, the capture image of target host 
phase, determines the number of disks on the 
target host, the number of partitions on each 
disk, and then proceeds to capture (andufsdump) 
compress (compress or gzip) each filesystem 
found. All disks, partitions, and compressed 
image file sizes are logged during the process. 
These logs will be used during the following 
“burn” and “restore” phases. 

Phase 4, the burn image to optical media 
phase, creates a bootable volume and any addi- 
tional volumes required for holding the com- 
plete system image. If the optical media is CD- 
R and the entire set of compressed images from 
the target host exceeds 650 MB, then the appro- 
priate number of additional CD-R volumes will 
be created. If the optical media is DVD-R and 
the entire set of compressed images from the 
target host exceeds 3.9 GB, then the appropriate 
number of additional DVD-R volumes will be 
created. 

Phase 5, the restore image phase, involves restor- 
ing the system image to the same or equivalent 
target host via bootable media. The boot pro- 
cess of the restore proceeds through a highly 
customized JumpStart process located on the 
bootable media itself. If the entire system 
Image is contained on more than one volume, 
then an additional CD drive or DVD drive 
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(depending on which optical media is being 
used) will need to be attached to the host 
receiving the image. 


Commands for the Five “CART” Phases 


The commands to invoke each of the five phases 
of the CART are listed below: 

Phase 1: Pre-Imaging Preparation of “Image 
Disk”. The command for this phase is entered 
on the “‘control host” and can be run prior to 
connecting the “control host” to the “target 
host”’: 

prepare _image disk 

Phase 2: Setup of Target Host. The command for 
this phase is entered on the “target host” after 
installing the “‘setup”’ floppy diskette, CD, or 
DVD: 


setup 


Phase 3: Capture Image of Target Host. The com- 
mand for this phase is forked on the “target 
host” by the Setup phase (if the setup process is 
successful): 


capture_system_image 


Phase 4: Burn Image to Optical Media. The com- 
mand for this phase is entered on the “control 
host”’: 


burn_system_image 


Phase 5: Restore Image. The command entered at 
the open boot PROM ‘ok’ prompt on the origi- 
nal “target host’? or a similar host that is to 
receive the image: 


ok boot cdrom 
or 
ok boot dvd 


(The second option currently being developed 
by Sun Microsystems as a standard option in 
the OBP.) 


A Closer Look at the Solaris Install CD 


The Solaris Install CD was reverse-engineered to 
determine how it boots and goes through its JumpStart 
process. The first area looked at was the logical layout 
of the Solaris Install CD itself. Table 1 reports this lay- 
out, specifically for the Solaris 2.6 Install CD. 


The first volume in the set of media containing 
the system image of the target host will emulate the 
Solaris Install CD layout above with three minor mod- 
ifications. First, “‘slice 0” is trimmed down to contain 
only the bare essentials to have the CD boot and run 
through the customized JumpStart process. Second, a 
seventh slice, “‘slice 6,’ is added to the CD. This slice 
will contain the log files generated during the image 
capture phase. Third, an eighth slice, “‘slice 7,” is 
added to the CD to hold some (if not all) of the com- 
pressed image files from the “‘target host.’’ The emu- 
lation of the Solaris Install CD with these 
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modifications is captured on the “‘image disk”’ and is 
reported in Table 2. 


Contents 


Installation and 
Distribution 
Miniroot 

Boot Info, sun4c* 
Boot Info, sun4m* 
Boot Info, sun4d* 
Boot Info, sun4u* 


Table 1: Solaris Install CD Layout. 

* Note that these slices contain the boot information 
(boo tblock) for the various hardware architectures of 
Sun Microsystems’ products that run Solaris. Also 
contained in these slices is the file .SUNW-boot-redi- 
rect which contains a single byte, the character ‘1’, to 
direct the firmware boot PROM program to look for 
the kernelon “slice 1” of the boot device. 


Slice _— Partition 





Contents 


Installation and Dis- 
tribution 

Miniroot 

BootInfo — sun4c * 


Slice Part. 


BootInfo — sun4m * 
BootInfo — sun4d * 
BootInfo — sun4u * 
Log Files 
Compressed Image 
Files 





Table 2: Image Disk layout. 
* The contents of these slices are as described in the 
note in Table 1. 
Also note that the CART implemented an 18 GB disk 
drive for the “image disk.”’ The size of the “image disk 
(or disks)”’ is determined by the size of disk space on 
the target host. 


Location of the JumpStart files on the Solaris Install 
CD and the “CART” 


During the “pre-imaging preparation”’ (Phase | ), 
select portions of the Solaris Install CD are copied to 
the “image disk.”’ Two specific portions are copied: 
(1) all of “slice 1,”” which contains the miniroot (the 
operating system); and, (2) parts of “‘slice 0,” which 
contains the pertinent files to allow the JumpStart 
mechanism to function. The location for these files is 
as follows: 


Ps0rSolaris 2.6/7 Tools) Boot) usr) 
Sbin/anstadl “d/anstali. conti p 


The pertinent standard issue JumpStart files are the fol- 
lowing: 
rules.ok Install JumpStart ““RULES”’ file 
install begin Install JumpStart “‘BEGIN”’ script; 
called out by the “‘rules.ok”’ file 
devsyn_finish Install JumpStart “FINISH’’ script; 
called out by the “‘rules.ok”’ file 
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How the “CART” Customized JumpStart 


Since, the JumpStart files (listed in the section 
above) now exist on the “image disk,’’ and the tool 
runs as ‘root’ user, the permissions on the location and 
the files themselves can be modified. Once the permis- 
sions are changed, the files can be modified (replaced) 
with highly customized versions, which is precisely 
the approach taken by the CART to highly customize 
the rest ore (Phase 5). 


The customization involved replacing the 
“rules.ok” file and adding a custom JumpStart BEGIN 
script, named “restore _system_image begin,” to the 
JumpStart location on the “image disk.”” The CART 
did not use the standard issue JumpStart files 
“install begin” and “‘devsyn_ finish.” Furthermore, 
the CART did not need a custom JumpStart FINISH 
script at all. All the actions the CART needed to 
accomplish were performed in the custom BEGIN 
script “restore _system_image begin.” These actions 
included verifying all of the appropriate disks are pre- 
sent, the disks then get partitioned, filesystems get cre- 
ated, and one by one, the filesystem images get 
restored. As additional volumes containing the system 
image are needed, the user is instructed to remove the 
current volume and insert the next. When all of the 
filesystems are restored and an appropriate bootblock 
has been installed, the customized BEGIN script 
reboots the system. When the ‘‘target host’’ finishes 
rebooting, it will look like the original host. The 
reboot at the end of the customized BEGIN script also 
has another side effect, it aborts the rest of the Jump- 
Start process. Thus, the system does not try to load the 
distribution, packages, nor patches. Instead, the com- 
pleted process looks more like an Ignite-UX “golden 
image” restore. 


Functional Details of the Five “CART” Phases 


For those who want to know the nitty-gritty 
details of the entire CART process, the functional 
details for all five phases are itemized below. 


Phase 1: Pre-Imaging Preparation of the “Image 
Disk” 
1. Requests user to perform physical setup 
2. Unmounts the ‘“‘image disk” 
3. Partitions the ‘‘image disk’”’ 
4. Creates filesystems 
5. Makes mount points for mounting the ‘image 
disk” 
. Mounts the ‘“‘image disk” 
. Places proper permissions on the “‘image disk’”’ 
8. Copies the contents of the “‘Solaris Install CD”’ 
to the “image disk” 
9. Removes the “‘lost+found” directories from 
slices sQ-s7 
10. Installs the various architecture bootblocks 
11. Opens the permissions on the JumpStart loca- 
tions in slice sO on the “‘image disk”’ 
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12. Creates additional directories with proper per- 
missions in the miniroot filesystem (slice s1) of 
the “image disk”’ 

13. Share the “image disk”’ and other local shares 
for the NFS mounts from the target host 


Phase 2: Setup of Target Host 


. Creates the work directory on the target host 

. Saves the original /etc/hosts file 

. Reassigns a temporary IP Address 

. Determines the network interface type 

. Saves the original network interface settings 

. Creates a virtual network interface 

. Establishes a network connection between the 
target host and the master control host 

8. Logs all changes so the original configuration 

can be re-established when complete 
9. Forks the execution of the ‘“‘capture image”’ 
script 


NYDN BWN — 


Phase 3: Capture Image of Target Host 


1. Determines the physical device name of the 
optical media (CD or DVD) drive 

2. Saves /etc/vfstab file information 

3. Determines the number of disk drives on the 
system using format 

4. Saves the partition information of each disk 
drive using prtvtoc 

5. Performs ufsdumps of the individual filesystems 

6. Compresses the filesystem dump files 

7. Calculates the number of media volumes that 
will be required to hold the entire image (com- 
pressed dump files) of the system 

8. Places the appropnate bootblock on the ‘“‘image 
disk” 

9. Copies the customized JumpStart BEGIN script 
to the JumpStart location on slice sO of the 
“image disk” 

10. Copies the customized “rules.ok”’ file to the 
JumpStart location on slice sO of the “image 
disk” 

1]. Restores the original /etc/hosts file 

12. Restores the original network interface settings 


Phase 4: Burn Image to Optical Media 


1. Kills the volume management if it 1s running 

2. Checks that the “‘image disk”’ 1s mounted 

3. Reads the number of volumes to create from 
the log files 

4. Puts files in ISO-9660 format 

5. Creates (burns) the bootable first volume 

6. Writes a label on the first volume indicating 
Volume | of ‘x’ number of volumes and target 
host name 

7. Creates (bums) all additional volumes 

8. Writes a label on each volume indicating Vol- 
ume ‘y’ of ‘x’ number of volumes and target 
host name 


Phase 5: Restore Image 


1. The target host to receive the image 1s booted 
from the first volume of the optical media set 
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2. Customized JumpStart located on the first vol- 
ume of the optical media automatically gets in1- 
tiated 

3. Mounts slice 6 of the optical media to /log_ dir 

4. Uses ‘fmthard’ to recreate the VTOC of the 
original disks to the target host disks 

5. Creates new filesystems on the appropriate 
slices of the new disks to match the filesystems 
from the original disks 

6. Mounts one at a time the intended slices on the 
new disk to the mount point /mnt 

7. Uncompresses and ufsrestores the compressed 
image files 

8. Places the pertinent bootblock on the boot 
device 


Handling the CD Mounts During Installation 


This paper would be remiss if it did not describe 
the manner in which the CART mounts the bootable 
first volume upon a restore of the image (Phase 5). 
The mounting of the bootable CD is identical to how a 
Solaris Install CD gets mounted during an install. This 
information is important because it elucidates one of 
the limitations of the CART, that being that if the 
CART produces a multi-volume set during the burning 
of the media (Phase 3), then an additional CD drive 
will be required during the restore of the image (Phase 
5). 

The details on how the contents of the bootable 
CD get mounted at boot time, 1s shown in Table 3. 


/tmp /tmp 

/proc /proc 
/devices _/tmp/devices 
/dev /tmp/dev 


/ /devices/pci@ 1f,0/pci@1,1/ 
ide@3/atapicd@2,0:b 
/devices/pci@ 1 f,0/pci@ 1,1/ 
ide@3/atapicd@2,0:a 
/dev/fd fd 


Table 3: The boot-time mounts of the CART bootable 
CD. 


| /cdrom 





Note: The information in this table is for an Enterprise 
250. The / and /cdrom mounts will vary depending 
upon the hardware platform being used. 


Table 3 reports that the miniroot (slice b on the 
optical media) represented by the line ending with a 
‘*:b,”’ is being mounted to the / root mount point. The 
miniroot is the operating system being used for the 
restoration (in the case of the CART) or the installa- 
tion (in the case of the Solaris Install CD). Since the 
operating system must always be present, the first vol- 
ume must always be mounted during the entire restore 
(or install) process. Thus, a second media (CD or 
DVD) drive must be attached to the “target host”? in 
order to restore a system image contained on a multi- 
volume set. 
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When booting from a CART generated bootable 
first volume or from a Solaris Install CD, the CD will 
get mounted as in Table 3. 


Optical Media 


The optical media used by the CART is CD and 
DVD. 


CD technology is attractive because it is inex- 
pensive. DVD technology 1s attractive because it can 
hold much larger capacities. 


After speaking with several vendors [10, 11, 12, 
14], the same message was echoed that there is really 
only one vendor, Pioneer, that provides a DVD writer. 
The vendors that provide CD/DVD recording solu- 
tions (hardware & software) use the DVD writer spec- 
ifications from Pi oneer. 


In terms of the writable media, there is a dra- 
matic increase in cost going from CD to DVD: “$1 per 
650 MB CD vs. ~$35 per 3.95 Billion Bytes DVD 
(which is not really 3.95 GB in the computer sense of 
the word but rather in the true mathematical sense of 
the word — which is a marketing ploy; see the note 
under the “DVD Technology Capacity and Compati- 
bility”’ section below). 


So far, Young Minds, Inc., (YMI), a leading CD 
recording technology vendor, is the only vendor I 
found working on a DVD solution for UNIX. In early 
2000, YMI was awaiting hardware and specifications 
from Pioneer. Furthermore, the current CD solution 
from YMI is scalable to handle DVD; all that is 
required is a DVD writer and a PROM update to the 
YMI CD Studio hardware. 


Unlike the CD technologies, where the CD-R, 
CD-ROM, and CD-RW versions all have the same 
capacity, each version of DVD technology has a dif- 
ferent capacity. See tables below. The varying capaci- 
ties of DVD technology will become a source of con- 
fusion for the consumer primarily because the various 
DVD technologies will not be compatible with one 
another. Currently, for non-video applications, the 
DVD technology market still needs to sort itself out. 
On the other hand, the CD technology market is well 
established. 


To learn more about CD and DVD technology, 
turn to the web [7, 8]. 
CD Technology Capacity and Compatibility 


Currently, there are four CD versions: CD-R, 
CD-RAM, CD-ROM, and CD-RW. All have the same 
capacity and are compatible with one another: 


Capacity | 
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DVD Technology Capacity and Compatibility 


The DVD standards committee is still trying to 
decide on the DVD standards [7, 14]. Currently, there 
are five DVD versions: DVD-R, DVD-RAM, DVD- 
ROM, DVD-RW, and DVD+RW. These versions have 
varying capacities, and thus, are mostly incompatible 
with one another: 


Capacity 
3 .95-4.7 Billion bytes 
2.6-4.7 Billion bytes 


DVD-R 

DVD-RAM 
DVD-ROM — 4.7 Billion bytes 
DVD-RW 4.7 Billion bytes 


DVD+RW 3 .0-4.7 Billion bytes 


As an aside, these capacities are truly in billions 
(1,000,000,000) of bytes and not a Gigabyte (GB) in 
the computer sense of the word, where a GB is defined 
as 1024 x 1024 x 1024 or 1,074,790,400 bytes. 





Where Do We Go From Here 


There are a couple of areas that the author would 
like to explore and develop in the very near future. 
First, modify the CART to work in a networked envi- 
ronment. Second, develop similar tools for other *NIX 
operating systems. 


Implementing the “CART” in a Networked Envi- 
ronment 


To implement the CART in a networked environ- 
ment running NIS or NIS+ and NFS would have been 
much simpler. The “‘control host’? could be placed on 
the network, and if a naming service is running, then 
all hosts will know about each other. Three types of 
CART servers can be placed on the network: (1) a 
standalone CART server; (2) a JumpStart server pro- 
viding only CART functionality; and, (3) a JumpStart 
server providing both CART functionality and Jump- 
Start functionality. A method for altering the BEGIN 
script to perform the restore (clone) can be imple- 
mented fairly easily. Thus, the JumpStart server can be 
used to perform site-specific JumpStarts as well as the 
CART method of imaging. Instead of maintaining the 
images of “‘target hosts’”’ on optical media, the images 
can be stored on network disk space and accessed 
through NFS. Again, this would be an entirely differ- 
ent way to unleash the power of JumpStart. 
Implementing the ““CART” in Other *NIX Operat- 

ing Systems 

One of the obvious next steps is to investigate 
implementing this tool in other *NIX operating sys- 
tems such as HP-UX, IRIX, AIX, Digital UNIX, 
FreeBSD, BSDI, and Linux. The first operating sys- 
tem to tackle on the list will be HP-UX, since it 
already has the fully developed Ignite-UX utility. 


Limitations of the “CART” 


1. The biggest limitation to the CART is encoun- 
tered when the target host contains relatively 
large filesystems (greater than 1 GB) and CD-R 
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is chosen as the optical media to store the sys- 
tem image. The limitation lies in the maximum 
carrying capacity of a CD-R which is 650 MB. 
By using compression utilities (compress or 
gzip) in this tool, and assuming about 60% 
compression efficiency, the maximum filesys- 
tem size that can be captured is about 1000 MB 
or | GB. So, when capturing to CD the system 
image of a target host that contains at least one 
filesystem larger than 1 GB, the CART will 
report and error. The error is due to the resul- 
tant compressed file for the filesystem exceed- 
ing the carrying capacity of the CD. However, 
there still is hope for systems containing 
filesystems larger than 1 GB. The system image 
could and should be captured to DVD-R, since 
the carrying capacity of a DVD-R media is 
about 4 GB. Again, assuming about 60% com- 
pression efficiency, a filesystem on the order of 
about 7 GB can be imaged. 

2. A system image saved to a multi-volume set 
requires an additional optical media drive for 
the restore operation to work. 

3. The system being restored must not only have 
the same kemel architecture but also the same 
platform architecture as the system that was 
imaged. As an example, even though an Enter- 
prise 250 and an Enterprise 450 have the same 
kernel architecture (sun4u), they have different 
hardware platforms. When the image from one 
is restored onto the other, the resultant system 
will not be bootable. 

4. When restoring a system or cloning to a like 
system, the disk drives in the host being 
restored must be as big or bigger than the disk 
drives in the original host imaged. 

5. Currently, the CART cannot capture system 
images of target hosts that have mirrors or 
RAID volumes (created by Veritas Volume 
Manager or Solstice Disksuite) on disk drives 
larger than | GB. When such logical volumes 
exist, the entire disk must be captured using the 
dd utility to ensure the private region on the 
disk is captured. By having to capture the entire 
disk as one image file, even after compressing, 
the resultant file would not fit on one CD vol- 
ume. This limitation goes away if the CART is 
implemented in a networked environment. 

6. Currently, the CART is only implemented for 
the Solaris operating system. 


Additional By-Products That Arose From _ the 
“CART” or Its Technology 


Several by-products resulted from the develop- 
ment of this tool. The possibilities are wide open for 
many more. The more useful of these by-products are 
identified below and how they were accomplished. 
Make Copies of the Solaris Install CD. How many 

times have you not been able to locate your 

Solaris Install media because there were too 
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few to begin with and they are all checked out 
(or locked in the office of the other system 
administrator)? Well, now you can make as 
many copies as you like (keeping in mind copy- 
right laws). Make a slight modification to the 
Boume shell script used during Phase | that 
copies select portions of the Solaris Install CD 
to the “‘image disk,” to have the script copy the 
entire CD. (Actually, during the initial stages of 
developing the CART, the original script copied 
the entire Solaris Install CD to the “image 
disk’”” anyway.) Following the completion of 
the copy, go straight to the burn phase (Phase 
3). While the ‘‘image disk” is in this state, as 
many copies that are desired can be burned off, 
one at a time. Using the CART configuration 
described earlier in this paper, it took about 30 
minutes to copy the entire Solaris Install CD to 
the “‘image disk” and about five minutes to 
burn off a copy. 


Make A Customized Solaris Install CD or CD 


Set. Instead of having the generic Install CD as 
provided by Sun Microsystems, place some of 
your Customized JumpStart configurations on 
the CD. This essentially makes your JumpStart 
come off of the CD instead of a JumpStart 
server. Who would want to do this? How about 
an experienced JumpStart administrator who 
finds him/herself in a non-networked environ- 
ment and is faced with a task of rolling out a 
large number of host installs or upgrades. If 
there are several additional packages and tar- 
balls needed for your intended JumpStart, addi- 
tional CDs may be required. By using the same 
technology incorporated into the CART, having 
additional CD volumes will not be a problem. 
However, just like with the CART, an addi- 
tional CD drive will be needed to perform the 
JumpStart from CDs. 


Make A Specialized Bootable CD. An industry 


authority on UNIX Backup and Recovery, (W. 
Curtis Preston) has asked to have a bootable 
CD built that will contain enough of an operat- 
ing system to allow third party backup and 
recovery software (Legato Networker and Veri- 
tas NetBackup) to run. Very customized scripts 
automatically invoked on the CD will prompt 
the user for certain information. Based on the 
information provided, the backup software 
selected will proceed to perform backups over 
the network to restore a complete system 
image. The intent of making the CD bootable is 
for the situation of a boot disk crash. For such 
situations, this specialized tool will perform a 
bare-metal recovery (disaster recovery on a vir- 
gin disk drive) and will result in bringing the 
system to the state of its last good backup. The 
development of this tool is currently in progress 
and looks quite promising. 
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Resources 


The following freeware products from Joerg 
Schilling [9] were used in the development and imple- 
mentation of the CART: 


cdrecord A program for creating single/mul- 
tiple session CD-R on a SunOS, 
Solaris, Linux, *BSD/SGI, HP-UX, 
AIX, NeXT-Step, or Apple-Rhap- 
sody system. 

sformat A program to format/analyze/repair 
SCSI hard disks on a SunOS, 
Solaris, or Linux system. 

sce A driver to send any SCSI com- 
mand to any SCSI device on a 
SunOS or Solaris system. 

fbk A driver to mount a file containing 
a filesystem; (File simulates Block 
device on Solaris). 

mkisofs Puts files in ISO-9660 format. 


A “Smart and Friendly” CD-RW 426 Deluxe 
CD-Recorder was used in the development and final 
implementation. No issue arose with the use of this 
CD-RW device during the development and use of the 
CART. 


Other CD-R recording hardware and software 
products (i.e., Young Minds, Inc., HyCD, Gear to 
name a few) could have been integrated into the 
CART as well. However, the price of “‘cdrecord”’ and 
its associated products could not be beat. There were 
not any issues encountered with the installation or use 
of the “cdrecord” products or with the use of the 
“Smart and Friendly” CD-RW 426 Deluxe CD- 
Recorder. Both of these products receive a high 
endorsement from the author. 


Acknowledgements 


First, I thank the Collective Intellect of Collec- 
tive Technologies who provided an_ invaluable 
resource and wealth of knowledge on many areas dur- 
ing the development of the CART. 


Second, I thank Joerg Schilling. Indispensable in 
the creation and final product of the CART were sev- 
eral shareware products provided by Joerg Schilling. 


Third, I thank Adelaida Esquivel, Senior Com- 
puter Programmer, who spent countless hours support- 
ing the process of writing and provided the proofread- 
ing of this paper. 


Author Information 


Lee “Leonardo” Amatangelo was graduated 
from the University of California, Irvine in 1983 with 
a B.S. in Molecular Biology and in 1985 with a B.A. 
in Anthropology. He has been working in the com- 
puter industry since 1981. Currently, he is a systems 
management consultant specializing in Solaris and 
disaster recovery for Collective Technologies. He can 
be reached via email at leonardo@colltech.com and by 


227 


Unleashing the Power of JumpStart Amatangelo 


physical mail at Collective Technologies, 9433 Bee 
Caves Road, Building III, Austin, TX 78733. 


References 


[1] Sun Microsystems, Solaris 2.6 -— Solaris 
Advanced Installation Guide, Mountain View 
CA, Part No. 802-5740-10, August 1997, Revi- 
sion A). 

[2] Heiss, J., “Enterprise Rollouts with JumpStart,”’ 
LISA XIII Conference Proceedings, 1999. 

[3] Kasper, P. A. & McClellan, A. I. Automating 
Solaris Installations — A Custom JumpStart 
Guide, SunSoft Prentice Hall, 1995. 

[4] Nemeth, E., Snyder, G., Seebass, S., & Hein, T., 
UNIX System Administration Handbook, 2nd 
Edition, Prentice Hall, 1995, Ch. 9. 

[5] Shaddock, M. E., Mitchell, M. C., & Harrison, 
H. E., “How to Upgrade 1500 Workstations on 
Saturday, and Still Have Time to Mow the Yard 
on Sunday,” LISA LX Conference Proceedings, 
1995. 

[6] Zuberi, A., “JumpStart in a Nutshell,’ Inside 
Solaris, February 1999, Ch. 1 

[7] http:/dvddemystified.com/dvdfaq.htm] . 

[8] http ://www.fadden.com/cdrfaq/faq00.html#[0-1 ] . 

[9] http://www. fokus.gmd.de/research/cc/glone/ 
employees/joerg_schilling/private/ . 

[10] http://www. gearcd.com/ . 

[11] http:/Awww.hycd.com/ . 

[12] http://www.pioneerusa.com/ . 

[13] http://www.smartandfriendly.com/. 
[14] http://www.ymi.com/. 


228 2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


A Linux Appliance Construction Set 


Michael W. Shaffer — Agilent Laboratories 


ABSTRACT 


Open source UNIX-like operating systems offer unique opportunities for administrators to 
create appliance style operating system and application packages that meet their own specific 
needs. This paper provides examples of several commonly useful appliance configurations based 
on the Linux operating system and documents the motivations, principles, and techniques behind 
the development of a basic "Linux Appliance Construction Set’ known as LxA which was used to 
produce them. Experience with LxA so far suggests that its use may help system administrators 
significantly reduce the amount of time spent deploying, maintaining, upgrading, and documenting 


certain types of hosts under their control. 


Introduction 


This paper details the philosophy, tools, tech- 
niques, and sample implementations that I have cre- 
ated in the course of developing a Linux Appliance 
Construction Set which I refer to as LxA. Certainly 
the philosophy and many of the the general principles 
detailed herein would apply to the construction of sim- 
ilar systems based on any open source UNIX-like 
operating system, but I chose Linux because of my 
long personal experience and familiarity with it. LxA 
expands to: Linux x Appliance, for instance: LRA 
abbreviates Linux Routing Appliance. 


The original design parameters for LxA included 
the requirement that the entire boot and root file sys- 
tem images fit on a single 3.5” floppy disk. The reader 
might legitimately wonder why I would embark on the 
creation of yet another mim Linux when there are 
already a number of excellent and similar projects 
active in the open source community [17, 11, 4, 10]. 
While most mini-Linux distributions aim to provide 
completely functional systems on low resource plat- 
forms, LxA attempts only to provide a single or small 
number of well defined features per configuration. It is 
not so much the minimal size as the minimalist philos- 
ophy that distinguishes LxA from its peers. In fact, 
some configurations such as LPA-CD (Linux Printing 
Appliance on CD-ROM) are quite large and require a 
CD-ROM for their root file system. Since the initial 
release of LxA, other community projects have arisen 
which pursue principles similar to those of LxA with 
apparently excellent results so far [5]. 


In general, LxA seems to be well suited to sys- 
tems such as: Internet connection sharing routers, fire- 
walls, bridges, routers, print servers, certain types of 
file servers (such as CD-ROM towers), terminal 
servers, point of sale terminals, browser kiosks, or in 
general any type of small to medium size single pur- 
pose system. LxA would probably not serve well for 
systems such as desktop ’power user’ or developer 
workstations whose users regularly install, test, and 
uninstall many applications. LxA is also not intended 
to serve as a general purpose or standalone Linux 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


distribution although it is hoped that the sample con- 
figurations will serve well as a basis for the future 
development of many different types of Linux appli- 
ances. 


Motivation 


As a system administrator, I frequently work 
part-time for small businesses, organizations, and 
departments who don’t for one reason or another have 
in-house system administration staff. When, in mid 
1999 I decided to move from South Carolina to Sili- 
con Valley, I was faced with the dilemma of support- 
ing a number of existing installations of Linux 
machines providing file, print, and network routing 
services which I would now be able to visit in-person 
perhaps only once a year. While Linux has inherited 
both a well deserved reputation for reliability and 
excellent facilities for remote administration from its 
UNIX ancestors, there are still situations which are 
extremely difficult to manage remotely. Furthermore, 
even with the recent surge in popularity that Linux has 
enjoyed it can still be difficult to find local companies 
or individuals who can provide reliable and affordable 
UNIX system administration to small customers in 
many regions. 


A typical serious problem that I considered fac- 
ing was the catastrophic failure of a hard disk or 
power supply in a critical machine such as a print 
server. While it would be easy to remotely coordinate 
procurement and installation of replacement hardware 
for such a failure, the rebuild or restoration of the soft- 
ware load for even a simple Linux server could be 
quite a dicey undertaking if trusted to inexperienced 
personnel working three-thousand miles away. I con- 
sidered the possibility of having replacement equip- 
ment shipped to my new location, configuring it 
myself, and then shipping it on to its destination, but I 
felt that this approach was undesirable since it would 
at least double the already significant costs, delays, 
and risks inherent in shipping computer sized pieces 
of electronic equipment around the mainland U.S. In 
addition, there was always the nsk that I might make 
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some trivial error in configuration that would only be 
discovered when the system was powered up in its 
final destination and failed to work properly (worst of 
all would be a mistake that also prevented me from 
remotely accessing the machine). 


In response to these considerations I formulated 
the idea of building Linux systems which would 
require no installation in the traditional sense but 
which would simply boot and run directly from the 
media on which they were delivered. My initial goal 
was to create fully functional internet connection shar- 
ing and print spooling systems contained entirely on a 
bootable floppy or CD-ROM. As I progressed in this 
endeavor, I also realized that I was unhappy with the 
significant amount of time that was required to 
‘adjust’ the typical Linux distribution to my own pur- 
poses and tastes after installation, and I decided to 
enlarge the scope of the LxA project to include devel- 
opment and documentation of a general purpose 
framework for building small Linux systems tailored 
precisely to the needs of my typical customers. Such 
Linux systems would allow me to provide customers 
with backup copies of the software load for their sys- 
tems by simply providing extra copies of their media 
upon delivery and would also allow me to deploy 
upgrades and fixes to systems much more economi- 
cally by simply shipping new CDs or floppies to all 
affected customers. Thus it was that I embarked upon 
the creation of LxA with the following goals: 

e Reduce system setup time 
e Reduce system maintenance and upgrade time 
e Reduce system complexity 
e Reduce probability and impact of hardware 
failure 
These are elaborated below. 


Reduce System Setup Time 


The typical procedure for setting up a new host 
using a general purpose operating system involves 
performing an installation from original media and 
then applying a long series of patches, packages, and 
procedures to adjust the system to the particular level 
of functionality and security desired. Certain operating 
system projects such as OpenBSD [14] seek specifi- 
cally to minimize the number of unnecessary compo- 
nents in a default installation and to dramatically 
reduce the number of steps required to achieve a high 
level of security after initial setup. Other projects such 
as Bastille Linux [1] do an excellent job of distilling 
and codifying community knowledge of the best prac- 
tices for a given platform into hardening scripts which 
aid administrators bo th in leaming and applying a con- 
sistent security policy to new systems. LxA, on the 
other hand, attempts to provide a set of procedures and 
examples for quickly composing new systems based 
on minimal filesystem trees that include exactly the 
components required in the proper configuration. 
Once a prototype system tree is created and tested, it 
can be deployed to new systems much more rapidly 
and with greater confidence in its correctness and 
security. 
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Reduce System Maintenance And Upgrade Time 


Many individuals and organizations seem to for- 
get that most of the time spent on any given comput- 
ing system will be spent maintaining it, not designing 
or installing it. For this reason, LxA seeks to reduce to 
an absolute minimum the time and problems involved 
in upgrading systems with new software. The typical 
LxA system will have its root filesystem and configu- 
ration files on one or more pieces of removable media, 
sO upgrading the system can be accomplished instantly 
and reliably by simply halting the system and replac- 
ing this easily handled media. 


The boot and configuration filesystems can be 
extensively tested before distribution to remote loca- 
tions to ensure a smooth upgrade, and in the event of a 
malfunction it is trivial to just replace the old media 
and restore the system to a working state until further 
troubleshooting can be performed. The desire to facili- 
tate this rapid and easy roll-forward and roll-back of 
system configurations was a primary incentive to the 
development of LxA. 


By avoiding completely the typical hard-disk 
based installation procedure, LxA_ systems also 
encourage rapid deployment of updated software once 
it has been tested, since mass upgrades and down- 
grades involve drastically reduced time and risk of ser- 
vice interruption. Additionally, the backup of the sys- 
tem binaries and configuration files of an LxA system 
becomes almost trivial since it means simply produc- 
ing and storing one or more copies of the boot and 
root disk images before distribution. This type of 
backup allows almost instantaneous recovery to 
known good states both from system failure and from 
system compromises or intrusions. Disaster recovery 
becomes simply a matter of obtaining suitable hard- 
ware instead of the time consuming and error-prone 
process of rebuilding a complete system from what- 
ever notes are available (or even worse, from mem- 
ory). 

Reduce System Complexity 


This was one of the primary motivations behind 
the early development of LxA systems. One of my 
original targets was to produce a working and useful 
Linux system composed of the absolute minimum 
number of pieces and, in the process, to gain an under- 
standing of exactly what each piece provided and 
required. A second motivation was to reduce dramati- 
cally the amount of time and effort needed to thor- 
oughly document the configuration and function of 
critical systems. All too often documentation is 
neglected or completely forgotten due to the time and 
effort required both to research and prepare tt. 


One can easily see that a Linux based firewall 
system such as LFA (Linux Firewall Appliance) com- 
posed of only around 40 files is far easier both to 
understand and to document than one based on a typi- 
cal general purpose distribution which might include 
hundreds or even thousands of files after even a 
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minimal installation. It is my belief that administrators 
are most likely to produce reliable and secure systems 
when they thoroughly understand all the components 
involved, and LxA seeks to facilitate this understand- 
ing by dramatically reducing system complexity. 


Reduce Probability And Impact Of Hardware Fail- 
ure 


A typical small LxA system such as LFA will 
often boot from a floppy and will run entirely from a 
RAM disk once started. In such minimal configura- 
tions, there is no need for either a hard drive or a CD- 
ROM. This obviously eliminates at least two compo- 
nents which can fail, and the omission of these devices 
will also aid the longevity of the system power supply 
by reducing the strain placed on it. For systems such 
as LPA-CD with larger filesystem space requirements, 
It is still possible for the system to operate without 
employing a hard drive, depending on the application. 


In some cases a fixed disk is the most efficient 
solution to certain requirements such as the need for 
swap or spooling partitions, but even in these situa- 
tions the system can still be upgraded without the need 
to manage the intemal storage. LxA systems typically 
feature some simple procedures in their startup scripts 
to optionally format, mount, and populate any neces- 
sary Swap or spooling areas at boot time, and LPA-CD 
includes examples for using RAM disks, network 
filesystems, or fixed disk partitions for these purposes. 
If desired, LxA systems can even partition their own 
fixed disks at boot time through the use of a non-inter- 
active partition editor such as sfdisk in their startup 
scripts. These features facilitate easy and rapid 
replacement of failed parts or even whole machines 
since LxA configurations can be easily tuned for no- 
install or self-install behavior on new hardware. 


Principles 


In pursuit of the goals stated above, LxA systems 
are generally designed to adhere to the following prin- 
ciples: 

e Build systems by composition, not reduction 

e Run from read-only and/or removable media 

e Omit login and run-time configuration 

e Use modern and standard software components 
The next sections detail these principles. 


Build Systems by Composition, Not Reduction 


This means developing a set of practices for 
identifying the minimal set of components required for 
any given capability and then adding exactly those 
components to a base or minimal LxA prototype 
image. 

Run From Read-only and/or Removable Media 


Although some LxA configurations make exten- 
sive use of fixed disk partitions for /var, /tmp, and 
swap areas, the general rule is to keep all system bina- 
ries and configuration files on removable and prefer- 
ably read-only media. The boot and root partitions 
may be made effectively read-only by loading them 
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into RAM disks at run time, mounting devices such as 
floppy disks in read-only mode, or using physically 
read-only media such as write-once recordable CDs. 
While floppy and CD-ROM media are readily accessi- 
ble and familiar to many users, some applications may 
benefit from the employment of alternative boot and 
root devices such as flash ram drives or cards. 


Omit Login and Run-time Configuration 


This principle not only helps to save valuable 
kilobytes on boot media, but also makes the system 
much more tamper-proof in potentially hostile envi- 
ronments. Obviously this approach also requires the 
administrator to perfect the system configuration dur- 
ing staging. This concept perhaps more than any other 
distinguishes LxA from other small Linux systems 
such as the Linux Router Project [11]. Certainly this 
approach will not work well for all applications, but a 
machine configured in this fashion is a true network 
appliance, running only exactly what it needs to per- 
form its designated functions. While console login 
facilities are often omitted in production LxA configu- 
rations, the example packages also include simple pro- 
cedures for installing a minimal set of interactive tools 
and a console shell on the system for troubleshooting 
during staging. 

Use Modern and Standard Software Components 


In contrast to other mini Linux distributions, 
LxA uses the most modern and standard components 
available wherever possible. The Linux Router Project 
(See [11]), for example, employs the Linux 2.0 kernel, 
libcS, and the BusyBox POSIX tools package [2] to 
provide a wide range of capabilities while maintaining 
the smallest possible disk and memory footprint. Cer- 
tainly there are a number of good reasons for using 
older components including smaller binary size, 
greater maturity, and often a more sedate pace of 
development. In fact, one of the greatest advantages of 


‘Discussions with LxA users during the writing of this pa- 
per led to the idea of building LPA systems based on a com- 
bination of a 48MB CompactFlash card, a CompactFlash to 
IDE adapter available from the Tuscon Amateur Packet Ra- 
dio organization (See [16]), and commonly available hot- 
swap hard drive carriages for standard size hard drives. This 
configuration should feature higher reliability than either 
CD-ROM or floppy based systems while still affording am- 
ple capacity for typical configurations. As of this writing, 
8-128 MB CompactFlash cards are available at reasonable 
price points. This option requires little modification to exist- 
ing LxA configurations since the TAPR device makes a 
CompactFlash unit appear to a PC system as a standard IDE 
drive, and the benefits of easy system upgrades will be pre- 
served since the removable hard drive carriage will make it 
easy to swap flash cards. Finally, this style of system would 
have the benefit of somewhat higher physical security than 
CD or floppy based configurations since many swappable 
drive carriages feature key locks, and the logistics of media 
distribution may even benefit slightly from the much smaller 
size and greater ruggedness of CompactFlash compared to 
floppy disks. Complete details of the outcome of this effort 
will be documented on the LxA project site [12]. 
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using open source tools in my experience is that stable 
versions of products tend to remain viable for a very 
long time due precisely to the availability of source 
code that can be compiled for older platforms even 
when binaries are no longer readily available. 


Nevertheless, I decided to avoid the use of older 
and non-standard components wherever I could. When 
pressed for space on small capacity boot media such 
as floppy disks, my approach was to save kilobytes by 
eliminating components entirely rather than recompil- 
ing, patching, or down-grading them. Usage of an 
older kernel and C library requires that all components 
be compiled specifically for that environment, and I 
wished to avoid forcing users to compile anything just 
to get a new LxA system running. Also, while older or 
customized components would undoubtedly save sig- 
nificant amounts of space on LxA systems, the bene- 
fits of using them would become completely negated 
if I wished to add a tool or service that depended on 
more modern and bulkier components such as glibc. 


This principle represents a significant difference 
in philosophy between LxA and several of the other 
mini distributions available. LxA provides not so 
much a specific distribution of Linux as a framework 
for assembling this sort of mini system from any cur- 
rent Linux distribution at hand. Using standard com- 
ponents allows more users to modify the system in 
unforeseen ways and should also allow LxA to track 
new developments in the Linux kemel, major libraries, 
and applications much more closely than distributions 
which rely on heavily patched or custom components.” 


Issues 


Remote Administration 


Since LxA configurations consistently use stan- 
dard binaries and libraries from more conventional 
Linux distributions, they frequently leave little space 
on small media such as floppy disks for remote or 
even local administration tools. The floppy based con- 
figurations such as LPA (Linux Printing Appliance) 
simply have no space for components such as sshd or 
inetd. In fact, in the case of LPA, there is no room for 
even a a minimal shell environment. In comparisons 
with other Linux distributions, even those of the mini 
variety, this could be considered a serious disadvan- 
tage for LxA. 


2Even as I was finishing this paper, I was working on up- 
grading all the LxA example configurations to linux kemel 
version 22.17, glibc 2.1.3, and the latest versions of all the 
binaries included. The total effort involved was a couple of 
kemel compiles, some work with find and cp to copy new bi- 
naries into the root filesystem trees, and re-making of the 
distribution tarballs. Other projects such as the Gibraltar 
firewall [5] have almost completely automated this process 
by employing scripts to find and assemble all required bina- 
ry components on the fly from the machine on which the 
configuration is performed. Such an option would probably 
be a valuable addition to LXA and may be added in the fu- 
ture. 
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On the one hand, I consciously decided up front 
to eliminate console and remote login capabilities 
wherever possible both for increased security and to 
free up disk space desperately needed for other things. 
On the other hand, for configurations such as LPA- 
CD, I have in fact included both a proper login shell 
and sshd? to allow the possibility for remote adminis- 
tration if desired. So, the lack of remote administration 
facilities can perhaps be seen as either an advantage in 
disguise or as a non-issue depending on the configura- 
tion and applications involved. In some repects 
addressing this issue may simply require an adjust- 
ment of perspective. LxA was intended to address pre- 
cisely those situations where remote administration 
capabilities are of little use, so what appears to be an 
advantage for other systems may in fact not be as sig- 
nificant as it seems in the environments for which 
LxA was designed. 


System Logs and Spool Areas 


The options LxA presents regarding boot-time 
management of fixed disks or the use of RAM disks 
for writable filesystems obviously raise the question of 
how to store persistent accounting information such as 
system logs. The typical LxA system is not one 
intended for hosting interactive logins, so it is 
assumed that most information of value would be 
included in the messages recorded by the system and 
kemel logging daemons. There is nothing to prevent 
an LxA system from using startup scripts that re-for- 
mat the system log partitions only when absolutely 
necessary, and this would allow persistence of system 
logs for later analysis and troubleshooting in the event 
of problems. Alternatively, in certain environments it 
may be desirable to have LxA based machines send 
their log messages to one or more remote syslog hosts. 
Remote logging is often an excellent method for facil- 
itating diagnosis of severe system failures or compro- 
mises and is commonly recommended in any event. 


Another issue which may arise is that use of the 
automatic partition and format features would result in 
the loss of any pending jobs on filesystems such as 
print spool areas. In general, LxA simply leaves it to 
the individual administrator to evaluate their applica- 
tion for a system such as LPA-CD and determine what 
balance of resiliency, persistence, and hardware 
requirements fits their environment best. For example, 
the following three scenarios might be considered 
when deploying print spooling and format conversion 
servers based on LxA: 

1. Booting from a CompactFlash device and using 
only RAM disks for spooling areas. This option 
would require careful consideration of the 
anticipated printing load, especially if a signifi- 
cant number of PostScript jobs would be 
spooled to non-PostScript printers. A configu- 
ration for moderate to heavy loads might bene- 
fit from the inclusion of 512-1024MB of RAM. 


3Sshd will be a standard component of LPA-CD by the 
time this paper is published. 
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This would allow an ample 128-256MB for 
working set and leave 384-768MB for a /var 
filesystem on a RAM disk. 

While this might at first seem like an extrava- 
gant amount of memory, consider that the result 
would be an entirely solid state print server that 
would likely perform extremely well, limited 
only by the speed of its CPU. Recent trends in 
hardware prices would make this only a moder- 
ately expensive configuration for most organi- 
zations. The obvious disadvantage of this con- 
figuration is that it makes no allowance for per- 
sistent spool areas, meaning all pending jobs 
would be lost in the event of a power failure or 
other serious event. In some cases this disad- 
vantage may well be outweighed by the 
ruggedness, reliability, and extremely low 
maintenance requirements that such a configu- 
ration would present. This would probably 
work well for distributed print servers that need 
to be scattered throughout a building, campus, 
or region with little on-site system administra- 
tion attention. 

. Booting from CD-ROM and using local hard 
drives for spooling, log, and swap area. This 
option is probably the easiest for the typical 
UNIX system administrator to evaluate and 
provision, and would probably work well in the 
a computer room or departmental environment 
where the desire is simply to have a standard 
and easily replicated configuration. Despite its 
conventional approach, this configuration still 
presents some options with regard to manipula- 
tion of the fixed disks at start up time. 

Simply mounting a pre-formatted and popu- 
lated /var filesystem provides for a simple 
startup and complete persistence but requires 
that spool disks be partitioned, formatted, and 
populated manually at setup time or in a sepa- 
rate machine. Automatic partitioning and for- 
matting, on the other hand, would provide for 
extremely quick setup and allow instant 
replacement of failed spool disks, and this 
might be a good choice if the cost of loosing 
spooled jobs and log files is felt to be out- 
weighed by the consequent reduction in mainte- 
nance effort. 

. Booting from CD-ROM and using NFS exports 
for spooling and log areas (with perhaps file 
based swapping over NFS as well). This option 
features a mix of advantages and disadvantages 
from each of the two options above. On the one 
hand, it relies on having a network infrastruc- 
ture already in place including at least one 
machine providing exported filesystems over 
NES. On the other hand, if such an infrastruc- 
ture is available, this option allows the simple 
startup and complete persistence of the second 
option, permits a completely solid-state hard- 
ware configuration for the print servers 
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themselves as in the first option, and doesn’t 
require either a huge amount of RAM or any 
local hard disks to be procured for individual 
print servers. The critical element to consider in 
evaluating this option is obviously the reliabil- 
ity and speed of the network and NFS file ser- 
vices available. Across a computer room, 
building, or even a small campus environment 
with widespread switched 100Mbit ethernet 
networks, this option could work nicely even 
for distributed print servers. An obvious disad- 
vantage of this configuration is that a failure of 
the spool area file server or its network segment 
could potentially cripple a large number of print 
services, but the same could also be said of the 
centralized print spooling servers used in many 
more conventional arrangements.‘ 


The main point the reader should note from these 
examples is that LxA by design makes choosing or 
even switching between all three of these alternatives 
almost trivial. Leaving aside the issue of hardware re- 
configuration for a moment, consider the effort that 
would be required to configure a typical Linux distri- 
bution for either the first or third options, and then 
consider that with LxA choosing between them means 
changing just a few lines in one startup script. The 
whole intent of LxA is not to dictate or assume a cer- 
tain pattern of operation but to make many different 
options available with as little effort as possible. 


Older Hardware 


While the low resource requirements of LxA sys- 
tems may allow the use of hardware that would other- 
wise be considered obsolete, it is important to note 
that older hardware is often more prone to failure and, 
more significant for many organizations, no longer 
supported by warranties or service agreements. On the 
other hand, the upgrade cycle for end-user systems in 
many organizations makes available a significant 
number of systems each year that are due for retire- 
ment. These systems increasingly often pose an 
expensive and troublesome disposal problem for their 
owners. Given the right application and software load, 
some of these machines can be successfully re-pur- 


posed as departmental, standby, or load-sharing 


4Upon reflection, the reader may reasonably ask what ad- 
vantage there could be in this option over a more conven- 
tional, centralized print server. After all, if anything, this 
option introduces more hardware rather than less into the 
picture when compared to using a single massive machine 
for the job. One example might be a situation where an or- 
ganization wishes to make a number of existing non-net- 
work and/or non-PostScript capable printers available for 
network printing throughout a building or campus. If a 
sufficient number of existing obsolete machines with net- 
work cards and CD-ROM drives are available, a configu- 
ration like this would allow upgrading these printers with 
both network direct and PostScript capabilities by con- 
necting them via serial or parallel ports to LPA-CD ma- 
chines distributed to wherever the printers physically re- 
side. 
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servers complete with a ready supply of spare parts 
from any unused machines in the pool to counter the 
greater potential for failure and lack of support. 


Tools and Techniques 


I decided when creating LxA to start with an 
empty system and add features through composition 
instead of trying to cut down an existing distribution 
or system through a process of reduction. This 
approach, I felt, would result in a smaller, easier to 
understand system with less development time. | 
wanted to create and document a system in which I 
could explain the purpose of every single file and pro- 
cess. In addition to learning more about Linux and 
UNIX systems through this exercise, I hoped to also 
develop a toolbox of general principles and techniques 
for building other minimal systems in the future. 


Assembling Services 


This is a general description of the tools and 
techniques I use to figure out the necessary compo- 
nents for a given service. I usually begin, of course by 
locating and examining the binaries for the service I 
want. For this example I’ve chosen to use Samba, the 
NetBIOS file and print service for UNIX-like systems. 
I always start by installing and configuring the desired 
service on a staging machine before I try to decon- 
struct it for a mini system. Of course, any available 
man pages, web sites, mailing lists, and other available 
documentation for the service in question should be 
considered required reading throughout this process. 


Distribution Package Managers 


Most Linux distributions employ some sort of 
package management tools for installing and remov- 
ing software. The two most commonly used package 
management tools are rpm (the RedHat Package Man- 
ager) and dpkg (the Debian PacKaGe manager). Distri- 
butions employing rpm include RedHat, SuSE, Man- 
drake, and many others. Debian, Corel Linux, 
Stormix, and a few others use dpkg. 


The maintainers of package files have usually 
put a great deal of work into determining the depen- 
dencies of each particular package, so I will start there 
if a distribution package is available for a particular 
service. The first step is to use the package manager’s 
"list files’ function to see exactly what files are 


# dpkg -1 ’*samba*’ 


Desired=Unknown/Install/Remove/Purge/Hold 
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installed on the system by a particular package. To 
query for the files installed by the rpm package for 
Samba, I would employ a command such as: 

## rpm -ql samba 

/etc/logrotate.d/samba 

/etc/pam.d/samba 

fete/re.d/init.d/smb 

/etc/smbusers 


[ier sages sanbe 


/usr/sbin/nmbd 
/usr/sbin/samba 
/usr/sbin/smbd 
/usr/sbin/swat 
/usr/share/swat 


/var/lock/samba 
/var/log/samba 
/var/spool/samba 


Newer dpkg based distributions will often split ser- 
vices and their clients up into several packages to min- 
imize dependencies and resource requirements for 
each package, so I start by querying the package man- 
agement database to see what installed packages might 
be related to Samba; see Listing 1. This reveals four 
packages, and I can tell from the descriptions that I am 
probably only interested in the first two for the sake of 
an LxA system. To query for the files installed by each 
Samba package, I would next invoke dpkg with the -L 
option for each interesting package: 


jf dpkg -L samba 
ES 
/usr/sbin/smbd 
/usr/sbin/nmbd 
/var/samba 


jeve/init.d/samba 
/etc/cron.daily/samba 
/etc/cron.weekly/samba 


df dpkg -L samba-common 
his 


/etc/samba/codepages 


| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed 
|/ Err?=(none) /Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) 


| |/ Name Version Descriprron 


+++ —_—— SS SSS SS Sa ss 6 eS SS SS SS LES SS SSS ss. S_——_—__ 





ii samba 2.0.7 73 
if “Samba=commom. 2..0u7 =3 
i: oSsamba=doe 2. O67 23 





=< ——— _S SS Ee ae ee 


A LanManager like file and print server for UNIX 
Samba common files used by both clients and server 
Samba documentation. 


pn task-samba <none> (no description available) 


Listing 1: Package management query: samba. 
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/etc/samba/smb.conf 


/etc/pam.d/samba 


In general, for the purposes of an LxA system, I 
ignore any files installed under /usr/doc, /usr/share/ 
doc, /usr/man, /usr/share/man, and /usr/local/man. 
These files are simply the documentation and man 
pages and are not necessary for functioning of most 
services. Other files which are probably not necessary 
are headers’ placed in_ /usr/include'— or 
/usr/local/Anclude. Files I give the most attention to 
are those placed in /etc, /bin, /sbin, /lib, /usr/bin, 
/usr/sbin, /usr/lib, /libexec, /usr/libexec, and /usr/share 
(outside of /usr/share/doc or /usr/share/man). Many 
services will install cron jobs (typically in 
/etc/cron.daily or /etc/cron.weekly), and these may or 
may not be necessary for an LxA system. Many times 
these cron jobs serve only to rotate logfiles, and such 
actions are not necessary for example when using a 
remote syslog host. On RedHat the files installed 
under /etc/logrotate.d may also be safely ignored for 
similar reasons unless the logrotate utility will be used 
on the mini system. In the example shown here, the 
RedHat Samba package includes the swat web based 
administration tool for Samba, whereas the Debian 
package seems to leave it to a separate installation. 
The decision to include or exclude utilities such as 
these will depend mostly on the size constraints of the 
boot media for the target system and the anticipated 
usefulness of the individual tools. Since I hardly ever 
use swat for managing Samba machines and it presents 
significant space requirements, I chose to omit it. 


The rpm and dpkg package managers, in addition 
to providing information about specific packages, will 
usually also provide a list of other packages which a 
service requires or suggests for proper operation. To 
query for the description and dependencies of a pack- 
age using dpkg: 

## dpkg -p samba 


Depends-~samba-common. (= 2057-3), 
iit bee (0-2 e272) a bnewrses 5. 


Once again, if the specified dependencies are not 
already present on the LxA base system image, they 
must be located and added. The Debian package man- 
ager lists dependencies in terms of other packages that 
must be installed, whereas the RedHat package man- 
ager may list either other packages or individual files. 
Both package managers provide version information 
for dependencies as well; stating whether each 
required package must be of some exact version or 
simply greater than a certain version for compatibility. 
If a dependency is itself a complex package, it may be 
necessary to repeat the process of exploration until all 
levels of dependencies have been satisfied. An exam- 
ple command for listing dependencies using rpm 
would be: 
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## rpm -q --requires samba 
pam >= 0.64 
samba-common = 2.0.6 
/sbin/chkconfig 
/bin/mktemp 

jist fpbin/ killa 
fileutils 

sed 

/bin/sh 

id enuse S62 2 
LEpe =. S646 

IA Dery pel sGe 

da Daal Ss Or.2Z 

iapas sion) 
libpam.so.0 
libreadline.so.3 
libtermcap.‘so~ 2 
/bin/csh 

/bin/sh 
/usr/bin/awk 
libe.so.6(GLIBC_2.0) 
(bic. sow 6 COLIBC 27..b) 


Both of the package manger listings included a 
dependency on a package named samba-common, so 
further exploration of this package would be necessary 
in this case. 


Although the package manager file listing 
reveals a large number of accessories and configura- 
tion tools, I know from experience with this service 
that the actual file and print service portion of Samba 
needs only two basic binaries to function, namely 
smbd and nmbd. Unfortunately, I have found that in 
making this sort determination, there is little substitute 
for experience and careful observation, although well 
documented services may prove easier to analyze than 
those which are not. As a final step before copying the 
binaries into my LxA image tree, I use the Is command 
just to check if they have any unusual permissions or 
modes set which may need to be preserved: 

4A 1s -al /usr/sbin/*mbd 
“EWE oR ox root root - 330097 

Jiu .27 -03956- /usr/sban/nmbd 
2LWAE RE OK eet OOt FOOT 7.3569: 0 

Jul 27 03256) /asr/sbin/smbd 


In this case there appears to be nothing out of the ordi- 
nary. 


Ldd 


After an initial survey using the package man- 
ager, man pages, and Is, I typically use Idd to see what 
system libraries the service components are linked 
with and where they are located: 


4A ldd /usr/sbin/*mbd 


/usr/sbin/nmbd: 
libdlsso.2-=>/lab/ la bdl soe? 
(O0x2aac3000) 
Liberyot«soa => /1iby/tiberype.soud 
(Ox2aac7000) 
lvberso.46. => ./ la b/ Libe-soc6 
(Ox2aaf4000) 


Jabs id= anise s0n.2: =) / aby 1d - 14a nix s6.2 


(Ox2aaab000) 
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/usr/sbin/smbd: 
libd bson.2 => o/iaby labdlso:22 
(Ox2aac3000) 
laberypte.so.l => /lib/liberyprssowl 
(Ox2aac7000) 
Libe :so.6 =>. /lab/libesso.6 
(Ox2aaf4000) 


/lib/ld-linux.so.2 => /lib/ld-linux.so.2 


(Ox2aaab000) 


If the libraries listed are not already present in the /lib 
directory of the LxA configuration I am working on, I 
would use something like: 


+ -ed /1ib 
## 1s -al libcrypt* 


-rw-r--r-- 1 root root 20340 Nov 1 
L356. a bery pr 2.27 SO 


lrwxrwxrwx 1 root root 17: Nov 9 


1903 ltbery ptaso 2 da berypt= 2 lee2 so 


# find liberypt* | cpio -pvd \ 
/usr/local/linutopia/root/1lib 
to add them. I recommend using find and cpio instead 
of cp since these tools usually do better job of preserv- 
ing permissions, modes, and linkages. 


Lsof 


If problems arise or I suspect that the service 
requires more than just the explicitly linked libraries 
shown by ldd, I will try executing lsof while the 
selected service 1s running to view a comprehensive 
list of file handles, sockets, and other resources the 
service has open. This command usually produces 
rather lengthy output, so I have snipped some trrele- 
vant details from the example in Listing 2. I observe 
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from this listing that both daemons are accessing 
another library which Idd didn’t say anything about, 
namely libnss_files. They also appear to have some pid 
files and such open in /var/samba and /var/log, so I 
would make a note to make these directories available 
in the /var filesystem of the mini system. Finally, they 
have some network ports open that they are listening 
on, namely netbios-ns, netbios-dgm and netbios-ssn. I 
would certainly take these into account when working 
out any ipchains rule sets for the mini system. 


Once the binaries, all required libraries, and any 
working directories are in place in the filesystem tree 
for the mini system, I would copy the /etc/samba 
directory contents over to the new /etc directory and 
modify them appropriately. When I really get stuck 
trying to figure out what obscure file or directory a 
daemon is missing in the mini configuration, I usually 
fall back on the old faithful strace utility. While this 
tool produces even more profuse and cryptic output 
than Isof, it is often the only means of identifying 
exactly what resources a process uses. Many utilities 
will open files or devices only briefly during startup or 
other phases of their operation, and this can sometimes 
foil efforts which rely solely on the snapshot view of 
system handles provided by Isof. The output from 
strace is a complete ‘diary’ of all system calls made by 
a process, and usually looking for calls to open(), 
stat(), read(), write(), socket(), connect(), send(), 
recv(), mmap(), and munmap() will reveal all the 
filesystem and network objects in use by the traced 
process. A typical invocation of strace would be: 


4A strace -o smbd.txt /usr/sbin/smbd -D 


/usr/sbin/nmbd 

(Vib Ad 2381230 

JLab jlbd soi?) sa 

(ia bi lapery pts 24l.2s.s0 

fli b/ Aa bes? a1. 22-80 

{/tip/ la bnes tt les=2.1).2 iso 
/var/log/nmb 

/var/samba/nmbd. pid 

*:netbios-ns 

*:netbios-dgm 
lazarus.hammes-chasn.com:netbios-ns 
lazarus.hammes-chasn.com:netbios-dgm 


/usr/sbin/smbd 

fla bpidld =o .1.22'so 

(Va b/ 1abdl=2.9..2)s0 

(Libs UIperypts2 1.2.66 

pla p/iabe =e 1 s2 eso 
jigb/labnss2tiles=2 1.2. se 
/var/log/smb 
/var/samba/smbd. pid 
“<netbros=sen (LISTEN) 


Listing 2: Using Isof to find resources used. 


# lsof | grep mbd 

nmbd 405 root txt REG 3,66 338092 41150 
nmbd 405 root mem REG 3,66 85238 4116 
nmbd 405 root mem REG 3,66 10224 4167 
nmnbd 405 root mem REG 3,66 20340 4163 
nmbd 405 root mem REG 3,66 936696 4119 
nmbd: 405 root mem REG 35:66° 32104 4178 
nmbd 405 root 3w REG 3,66 804 923745 
nmbd 405 root 4ww REG 3,66 20 878636 
nmbd 405 root 5u inet 3016 UDP 
nmbd 405 root 6u inet 3018 UDP 
nambd -405 root: Jusinet 3022 UDP 
nmbd 405 root 8u inet 3024 UDP 
smbd 407 root txt REG 3.66 738556 41149 
smbd 407 root mem REG 3.66 85238 4116 
smbd 407 root mem REG 3,66 10224 4167 
smbd 407 root mem REG 3.66 20340 4163 
smbd 407 root mem REG 3,66 936696 4119 
smbd 407 root mem REG 3,66 32104 4178 
smbd 407 root 3w REG 3,66 500 923748 
smbd 407 root 4ww REG 3,66 20° 87:30:37 
smbd 407 root 5u inet 3030 ECE 
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This example would write its output to the smbd.txt 
file. 


Source Code 


Of course, one should never forget the one 
unique resource available to all users of open source 
software, namely the source code. While reading the 
source code for system utilities undoubtedly requires 
the most significant investment in time and effort of 
any technique discussed here, it also ultimately yields 
the most significant and longest lasting results in 
terms of experience, understanding, and confidence 
when practiced regularly. Based on my own experi- 
ence, I steadfastly maintain that acquiring and practic- 
ing even basic programming and code reading skills 
will make any system administrator much more capa- 
ble, flexible, and confident when dealing with new 
systems, unfamiliar problems, and unusual configura- 
tions. 


Configuring a Read-only Root File System 


Actually, a workable read-only root file system 
for most Linux applications differs little from the typi- 
cal hard disk based root file system. LPA-CD uses a 
slightly different file system layout from many Linux 
systems, but that’s only for the sake of simplicity. The 
specific things that I found which needed attention 
included: 


The /var file system 


Many processes place run time files in /var. 
Even a completely CD based system that uses no con- 
figuration floppy will still require at least a small 
RAM disk mounted on /var so that processes like smbd 
have somewhere to save their PID and temp files. 
Usually I will make /tmp a symlink to /var/tmp to 
keep all writable areas under one mount point. When 
an LxA system is configured to format and mount a 
/var file system automatically at each startup, the nec- 
essary files and directories may be populated within it 
either by using tar to unpack a skeleton image or by 
running a simple script of mkdir, touch, chown, and 
chmod commands. LPA uses the latter approach since 
it has no room for the tar binary whereas LPA-CD uses 
the former to make modifications easier. 


Sockets in /dev 


A few processes, notably syslogd and Ipd, create 
sockets in the /dev directory to listen for their clients. 
Fortunately, syslogd poses little problem since it 
accepts a -p option specifying an alternative path for 
the socket that it usually opens at /dev/log. For LPA- 
CD, I typically start syslogd with a command line such 
as: 

# syslogd -p /var/run/syslog.socket 
I then create a dangling symlink from /dev/log to 
/var/run/syslog.socket in the read-only root file system 
to direct syslog clients to the real socket. 


BSD l|pd, on the other hand, presents consider- 
ably more trouble since it offers no command line 
argument to specify its socket path like syslogd. I was 
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forced in this case to break down and recompile the 
entire lpr package from source after changing a hard- 
coded #define in the lpr source code fiom /dev/printer 
to /var/run/Ipd.socket. Again, I created a symlink from 
/dev/printer to /var/run/Ipd.socket in case any clients 
outside the lpr package expected this socket to exist. 
When tracking down issues like this, I usually find Isof 
and strace to be indispensable. 


Various Files in /etc 


The only files in /etc I know of that a Linux sys- 
tem tries to write to under normal conditions are 
/etc/mtab and /etc/ioctl.save . I took the easy way out 
with mtab and just made it a symlink to /proc/mounts 
which the kemel maintains automatically. This change 
allows df and mount to work correctly, and the system 
will simply discard anything written to /proc/mounts. 
The ioctl.save file is read and written by init and is used 
to set console parameters when the system enters sin- 
gle user mode. While init normally creates this file at 
boot time, it does not actually require it and will func- 
tion perfectly well in its absence. All the other files 
typically found in /etc can either be read only or can 
just be symlinks to something on another file system if 
preferred. The one command that I haven’t found any 
way to use with a read-only /etc directory is passwd. 
Because of the way passwd shuffles files around while 
changing an entry in /etc/passwd, it doesn’t seem pos- 
sible to use it successfully unless it can write to all of 
the following: /etc, /etc/.pwd.lock, /etc/npasswd, and 
/etc/passwd. 


If the /etc directory will be populated with sym- 
links, then there are exactly two files which must be 
physically present in this directory in order for init to 
start: /etc/inittab and /etc/rc.init. The /etc/inittab file 
cannot be relocated unless init is recompiled. Addition- 
ally, /etc/inittab must specify the path to the system 
startup scripts, and these scripts must exist on the root 
file system since nothing else will be mounted at this 
point. In the case of LPA-CD, I created an /etc/rc.init 
script which mounts the media for the /local file sys- 
tem so that all the dangling symlinks in /etc become 
valid. After running this script, LPA-CD’s inittab 
hands off control to /etc/rc.local, which is actually a 
symlink to /local/rc.local, and this script handles the 
remainder of the system initialization. 


Boot Process and Initrd Image 


The Linux kemel contains support for a unique 
feature known as initrd or the Initial RAM Disk [9]. 
This capability is of critical importance to space con- 
stricted LxA configurations such as LPA since it 
allows a system to load and mount a small root 
filesystem image from a device that the kemel may 
not even contain drivers for. This mechanism requires 
only that the kernel have support for both RAM disk 
devices and initrd devices built in and places the bur- 
den on the boot loader to get the root filesystem image 
loaded and unpacked in memory along with the ker- 
nel. Thanks to this feature, the standard kernel for 


237 


A Linux Appliance Construction Set 


floppy based LxA configurations is able to defer sup- 
port for both floppy and ide devices to loadable kernel 
modules for a moderate savings in kemel size. The 
typical boot sequence for Linux systems using the ini- 
trd mechanism 1s: 
1. BIOS loads first stage boot loader from a mas- 
ter boot record into memory 
2. First stage loads and executes second stage 
boot loader 
3. Second stage boot loader (usually lilo or sys- 
linux) loads the kemel and initrd image into 
memory and uncompresses both 
4. Linux kermel executes the script /linuxrc on the 
initrd image if present 
5. Linux kernel creates the first userland process 
by executing /sbin/init 
6. The init process executes system startup scripts ( 
rc.init and rc.local for LxA) 


The original purpose for the initrd mechanism 
was to allow booting from devices such as SCSI disks 
using a standard modular kemel without support for 
these devices built in. The idea was that the /linuxrc 
script could be exploited to load required driver mod- 
ules from the initrd root filesystem image and then 
mount the real root filesystem from another device. 
LxA actually doesn’t use the /linuxrc script at all but 
instead just runs the whole system from the initrd 
image. For the floppy based LxA configurations, the 
standard script for producing an initrd image from a 
root filesytem tree 1s: 


#4! /bin/sh 

EM st. 37 disk/ initrd 

EM “@ a/disk/ianitrd. ez 

dd if=/dev/zero of=./disk/initrd \ 
bs=1024 count=s1 

losetup /dev/loopO ./disk/initrd 

mkfs.minix -i $2 /dev/loop0O 

mount /dev/loopO /mnt 


ed if Poot 
fand: = 3) epio;pud</mntW>: /dev/ null 
edi gus 


umount /mnt 

losetup -d /dev/loop0O 
gzip -9 ./disk/initrd 
exit 0 


Creating a Bootable CD 


Like most things PC, the mechanism for 
bootable CDs is a hack. While most PCs aren’t all that 
good at booting from CD-ROM drives, they are really 
good at booting from floppies, so that’s what they do. 
To make a CD bootable, I would first produce a floppy 
disk that boots the way I want it to. For instance, I 
would use a a script similar to the following: 

superformat /dev/£d0 

syslinux /dev/fd0 

mount -t msdos /dev/fd0O /mnt 

cp vmlinuz /mnt 

echo “prompt: 1) /mnt/syslinuxectie 
echo "timeout 100" >> /mnt/syslinux.cfg 
echo "default vmlinuz” \ 
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2? Imnt/ syslinux set g 
echo “append root=/dev/hdc" \ 

»> /mnt/syslinux.cfg 
umount /mnt 


Now, I would reboot my test machine using the 
floppy to see that it loads the kemel properly and 
mounts the correct root device (/dev/hdc in this exam- 
ple). If all seems well, then I would make an image of 
the floppy with dd using a command like: 


# dd if=/dev/fd0 of=boot.img 


The file boot.img should then be placed within the 
directory tree from which the ISO file system image 
will be created. Finally, the -b option of mkisofs will 
cause it to place a special entry in the ISO9660 catalog 
pointing to this floppy image file. What the PC’s 
BIOS actually will do at boot time is map this file on 
the CD to an ’emulated floppy’ and then try to boot 
from it as if it were a real floppy disk. The rest of the 
BIOS and any boot loader residing on the floppy 
image are usually completely fooled. Of course, once 
the Linux kemel is loaded, the whole illusion evapo- 
rates and the CD-ROM goes back to being an ordinary 
device like /dev/hdc. The following commands would 
be typical for creating the actual ISO image using 
mkisofs: 
# cp boot.img CD/boot/boot.img 
## mkisofs -v -R -b boot/boot.img \ 
-c boot/boot.catalog \ 
-o img/lpacd.iso CD 

Note that the path given in the -b option should be the 
path relative to the root of the CD file system. At this 
point the only remaining task would be using cdrecord 
to actually write the ISO image to a blank recordable 
CD: 


## cdrecord -v speed=4 dev=2,0,0 \ 

img/lpacd.iso 
This command is typical of what I would use on my 
own machine, but the speed= and dev= parameters are 
specific to the particular hardware in use. The -scanbus 
option to cdrecord will show what devices are available 
and what value to pass in the dev= parameter for each 
particular drive. The speed= parameter should be set to 
a value compatible with the capabilities of both the 
drive and media used. The mkisofs utility has a number 
of other options for specifying boot disk and kemel 
images when creating bootable images. Although the 
LxA example configurations have not yet been built 
for Sparc hardware, bootable ISO images for that plat- 
form can be produced as well. For more comprehen- 
sive details regarding the usage of mkisofs and cdrecord 
Se | 


One important choice that must be made when 
creating boot and root disks on recordable CD-ROM 
media 1s whether to use CD-R (write-once) or CD-RW 
(re-writable). Despite the obvious penalties for mis- 
takes when using a write-once media, I prefer to use 
CD-R unless I am completely sure that all target 
machines have CD-RW compatible drives installed. 
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Many older CD-ROM drives are not capable of read- 
ing or booting from CD-RW disks, but if the drive 
includes ’Multi-Read’ compatibility in its specifica- 
tions it should read either media with no problem. 
Using the Kernel Command Line 


In order to allow the maximum amount of flexi- 
bility for booting an LPA-CD image without requiring 
changes to the fc.init script bumed onto the CD, | 
decided to employ a couple of custom kemel com- 
mand line parameters to pass information to rc.init 
through the boot loader. The basic idea when using the 
kernel command line on Linux is to insert whatever 
parameters are required into the append= parameter of 
the boot loader and then take advantage of the Linux 
/proc/cmdline node to extract these parameters from 
the kernel at runtime. For the sake of LPA-CD, two 
parameters were required, the block device and the 
filesystem type for the /local filesystem. The com- 
mand line parameters are read from the file syslinux cfg 
by the syslinux boot loader at startup time and are spec- 
ified as follows: 

append root=/dev/hde lxa_lfs=vfat \ 
lxa_ldev=/dev/fdl 

These parameters are then passed on the the kernel by 
the boot loader and may be retrieved from the 
/proc/cmdline node with various utilities such as awk. 
The actual rc.init script used for LPA-CD and illustrat- 
ing this technique is shown in Listing 3. Any number 
of utilities such as cut, sed, or grep, or even perl can be 
used for this sort of task, but I chose awk since it was 
the smallest binary which provided a general purpose 
scripting and text processing tool available on my 
development system. 


Example LxA Configurations 


LRA: Linux Routing Appliance 


This system provides: a DHCP server, local and 
caching DNS, dial-on-demand PPP, firewalling, and 


#!/bin/sh 

PATH=/bin:/sbin 

export PATH 

## Mount /proc filesystem first 
mount -n -t proc proc /proc 


A Linux Appliance Construction Set 


masquerading for a local network. The entire system 
boots from a single floppy and runs from a RAM disk. 
The minimum hardware requirements are: a 486 CPU, 
16MB RAM, an ethemet card, a 3.5” floppy, and a 
modem. LRA was the first configuration developed 
and seems from analysis of the LxA site access statis- 
tics to be the most popular among users. 


LFA: Linux Firewall Appliance 


LFA requires little more than a floppy disk drive, 
an ethernet card, a CPU, and some RAM. Users of 
LFA have reported that it works adequately with as lit- 
tle as 6MB of RAM even with the interactive login 
package added. All of the work of this system is done 
by the kemel’s own firewalling and routing modules, 
and user reports suggest that LFA can function well 
for moderate traffic loads across one or two 10Mbps 
ethernet segments even with only a 486 CPU. Heavy 
loads, more segments, or 100Mbps ethemet would 
definitely require at least 16MB of RAM and a Pen- 
tium class CPU for satisfactory operation. LFA 1s, I 
believe, almost as minimal a Linux system as one 
could possibly assemble and still perform useful work. 
After the startup scripts have run the only processes 
besides the kemel and init on this configuration are 
syslogd and klogd. Because of the minimal process 
load, LFA is extremely stable, runs entirely from a 
RAM disk, and leaves little opportunity for compro- 
mise or failure. This system can provide firewalling, 
routing, and masquerading between two or more ether- 
net networks. For many small businesses or isolated 
departments, a commercial hardware or software fire- 
wall package may be excessively costly and present 
many features that would go unused. In situations such 
as these, LFA can provide a secure and reliable alter- 
native which supports many of the most useful fea- 
tures of a commercial firewall or router at much lower 
cost. Excluding the /dev directory, the root filesystem 
for this configuration contains only around forty files 
and directories; see Table 1. 


4 Parse custom command line parameters from /proc/cmdline 
eval “cae /proc(emdl ine: || awk: “(match (sd, (ian ite=| oj a7 ya. \ 


pranté ("XA LES=%s 


ax ports tka Leos vn, oN 
Ssubstrts.0,- (RSTART 4.8): 5. (CREENGTE. <= <6)) 0: 


pat 


eval “cat -/proe/emdline | ‘awk "4 mateh ($0, /ixa-ldev—=["]4/)=> \ 


printf ("LXA_LDEV=%s 


Ssubstr ($0. <CRSTART + 29)2 > “CREENGTHe +992) 


## Mount /local filesystem 


export bea BEV Vm \ 


he Tes 


echo. -m “Mounting “SLXA IEDEV* on //locad: as type: SLXA IRS) a." 


mount -n -t SLXA_LFS $LXA_LDEV /local 
echo ’done.’ 
exit O 


Listing 3: The rc.init script used for LPA-CD. 
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/bin | location for user binaries : 

/bin/sh | ash, an interpreter for startup and other scripts 

/bin/echo used for writing configuration settings into /proc filesystem nodes and 
displaying boot messages 

/etc location for system configuration and boot files 

/etc/inittab configuration file for /sbin/init 

/etc/fstab | list of active filesystems, their locations, and options 

/etc/hosts | local hostname to IP address mappings 

/etc/rc.init first setup script executed by /sbin/init —_ 


/etc/networks local network name to IP subnet mappings 


/etc/rc.local main system startup script 





/etc/services mapping of port numbers to service names 


/etc/group list of local user groups 
configuration file for GNU C library map functions such as gethostby- 
name() 
/lib | location for dynamically linked libraries 


/lib/ld-linux.so.2 
/lib/libc.so.6 


/ib/libnss_files.so.2 | name service switch module used by C library for lookups from files such 
as /etc/hosts ; 


the dynamic library loader, used by all dynamically linked executables 
the standard GNU C runtime library, huge but essential 


/\ib/modules location for kernel modules 
/lib/modules/* 
linuxrc optional pre-init script executed by the kemel at startup (not used by LxA) 


any required drivers for devices such as ethemet adapters 


/proc mount point for the kernel’s proc pseudo filesystem 


sbin location for system binaries 


[/sbinfwclock | used to sync system time with hardware clock 
[binfontig | configure network imerfces 
/sbin/ipchains | used to configure the kemel’s IP firewalling rules 

/sbin/mount | used to mount and remount /, /proc, and any other filesystems 

| /sbin/route | used to configure network routing tables 

/sbin/insmod used to load modules such as ethemet drivers 

_/sbin/syslogd system message logging daemon - 

| /sbin/klogd | kernel message logging daemon ] 
/tmp location for working and scratch files 

| /var | location for spool and log files 

/var/run | location for lock and pid files ; 


Table 1: Root filesystem. 
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Due to its extremely minimalistic nature, I have 
also found LFA useful as a baseline configuration for 
creating things like custom rescue and repair diskettes. 


LPA-CD: Linux Printing Appliance on CD-ROM 
System Requirements 


LPA-CD is a complete Linux system which will 
boot and run directly from the ISO image provided on 
the LxA project site. The minimal system require- 
ments for an LPA-CD system are: a 386 or 486 CPU, 
4MB RAM, an ethermet card, a floppy disk drive, a 
CD-ROM drive, and obviously at least one printer. 
Recommended but optional items include a fixed disk 
or Zip drive for temporary files and swap space and as 
much RAM as the budget will allow for better perfor- 
mance and higher load capability. While LPA-CD 
contains many more capabilities than LFA, it benefits 
slightly in terms of RAM requirements as a result of 
mounting its root file system from the CD-ROM 
instead of a RAM disk. The minimal configuration has 
proven to work well for a light load such as a dozen 
Windows clients printing small to moderately sized 
documents through client side PCL drivers to PCL 
printers. Situations involving large numbers of clients, 
large print jobs, or conversion of complex, color 
PostScript jobs will benefit from as much CPU, RAM, 
swap, and spooling space as the system can hold. Only 
evaluation of the anticipated system load can dictate 
what hardware configuration can be considered mini- 
mal for a given situation. 


Features 


An LPA-CD system will spool, convert, and out- 
put print jobs from either UNIX Ipr or Windows 
Samba clients. This package allows easy setup and 
management of print spooling hosts for mixed plat- 
form networks, LPA-CD 1s not based on any particular 
distribution, but the original sample configuration 
included components from the Debian 2.2 (Potato) 
system with the Linux kernel version 2.2.14 and glibc 
version 2.1.3. In the current release, LPA-CD includes 
the option for a minimal but complete set of interac- 
tive tools and a sulogin shell for use during staging. 
Since the system includes tools like vi, it is possible to 
save configuration changes to onto the local configu- 
ration floppy while the system 1s operating (something 
that is not possible with many of the more resource 
restricted configurations such as LRA, LFA, and 
LPA). This allows an LPA-CD box to be started and 
configured completely standalone if desired. Basic 
features in the current release of LPA-CD include: 

e Linux kernel version 2.2.17 

e Glibc 2.1 3 

e Linux ipchains kernel firewalling 

e PAM (Pluggable Authentication Modules) 

e Samba for serving Windows printing clients 

e BSD Ipr for UNIX printing clients 

e Ghostscript for conversion of PostScript/PDF 
print jobs to printer-native output 
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e Netpbm for conversion of various graphics for- 
mats to printer native output 

e Magicfilter for easy and flexible configuration 
of print job conversion based on file magic 
numbers 


LPA-CD uses the standard BSD Ipd for spooling 
of print jobs, and can support both directly connected 
serial or parallel printers as well as network attached 
devices such as HP JetDirect adapters. LPA-CD fills a 
unique application niche since it provides both print 
job spooling and automatic conversion of many job 
formats for UNIX as well as Windows clients. The 
TODO list for LPA-CD also includes adding support 
for MacOS clients via the Netatalk package in the near 
future. 


System Configuration 


An LPA-CD system is comprised of two disk 
images, a bootable CD-ROM and a bootable floppy. 
Either may be used to bootstrap the target machine 
since many older machines don’t have a BIOS capable 
of booting from a CD-ROM drive. The result is the 
same in either case; the root filesystem will be the 
1S09660 filesystem found on the CD, and all configu- 
ration files that would need to be changed locally 
reside on the floppy image which will be mounted on 
/local at system startup. LPA-CD offers two choices 
for a quick start; using disk images or using a tarball. 
If the environment will require only changing some 
configuration files, then the disk images are probably 
easier to start with. If, however, extensive changes or 
additions to the CD image are anticipated, then the tar- 
ball will prove more suitable. The CD image and boot- 
strap procedures are intended to work well for the 
majority of typical setups, and the tarball is probably 
only appropriate if the configuration requires unusual 
features or extensive changes. In either case, of 
course, either a CD-R or CD-RW drive is required for 
actually writing the ISO image to a CD-R disk. 


System Startup 


The rc.init script included on the CD image is 
responsible for mounting the appropriate filesystem on 
/local, and the /local/rc.local script actually does most 
of the work of starting up an LPA-CD machine. This 
script should be tuned for the specific machine on 
which it will be run. Only a summary description of 
the script is provided here. The phases of startup 
include: 

e Loading modules: Near the top of the file are 
several invocations of modprobe which load 
driver modules for devices such as the ethernet 
adapter. 

Preparing and activating the /var filesystem: 
After any required modules are loaded, there 
are several alternative sections in the rc.local 
script showing examples of how to create 
and/or mount various types of filesystems for 
use as temp and spooling areas. Only one of 
these alternatives should be used, and the others 
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may be removed or ignored. As mentioned in 
the Issues section, a careful evaluation of the 
resources available and the anticipated load 
should be made before choosing this option. 
Examples for each alternative are shown here: 
i## RAMDISK 
dd if=/dev/zero of=/dev/ram0 \ 
bs=1024 count=4096 
mkfs.minix -v -i 8192 /dev/ram0 
mount -t minix /dev/ramO /var 


## LOOPBACK IMAGE 

losetup /dev/loop0O /local/var.img 
mkfs.minix -v -i 8192 /dev/loop0O 
mount -t minix /dev/loop0O /var 


+: LOCA, DISK 
mkfs.minix -v -i 8192 /dev/hdbl 
mount -t minix /dev/hdbl /var 


if NFS 

modprobe nfs 

mount -t nfs \ 
spoolhost:/spool/linprinter /var 


## Unpacking /var filesystem image 
tar -C./ var -=xvzi-—\ 
/local/var. tar.gz 


The -V option to mkfs.minix causes a Minix V2 
filesystem to be created which is necessary for 
volumes over 64MB in size. The -i 8192 option 
causes allocation of 8192 inodes in the new 
filesystem which is a somewhat larger number 
than the default. If volumes of more than 
256MB will be used, then the ext2 filesystem is 
probably a better choice. 

Activating network interfaces: This section 
includes standard ifconfig and route invocations 
to set up the interfaces and routes as necessary 
for the particular environment. 

Configuring ipchains rulesets: The default 
ipchains statements included install a fairly 
strict set of filters allowing only the traffic nec- 
essary for lpr and Samba clients to connect. 
The rules also allow clients on the local net- 
work to ping and be pinged by the LPA-CD box 
to aid in diagnostics. 

Starting services: This section at the end starts 
syslogd, klogd, nmbd, smbd, and Ipd. 

Sulogin Shell 


The CD image contains a minimal but mostly 
complete set of tools for system setup, monitoring, 
and diagnosis.° 


‘In production configurations many of these tools may go 
unused, but they are included in the ISO image for LPA-CD 
by default since the cost to add them is relatively much high- 
er for a write-once media like CD-R versus a re-writable 
media such as a floppy. The floppy based configurations 
such as LRA include in their configuration scripts the option 
to add a login shell if desired. 

The only non-standard tool included with LPA-CD is cryptwd 
which is a small utility designed to help replace the ordinary 
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e Interactive/miscellaneous commands: awk bash 
clear cryptwd echo sh vi 

e File operations: cat cp chmod chown dd find grep 
less In Is mkdir mv rm touch 

e Filesystem maintenance: df dosfsck e2fsck fdisk 
fsck fsck.ext2 fsck.minix fsck.msdos fsck.vfat losetup 
mkdosfs mke2fs mkfs mkfs.ext2 mkfs.minix 
mkfs.msdos mkfs.vfat mkswap mount swapoff 
swapon umount 

e File archive maintenance: cpio gzip tar 

e System status: date dmesg free hostname Ismod 
lsof ps strace uname 

e System/process Control: halt hwclock insmod kill 
modprobe reboot rmmod setserial shutdown tunelp 

e Networking: ifconfig ipchains netstat ping route 

e Samba: mksmbpasswd smbclient smbpasswd smb- 
spool smbstatus testparm 

e BSD print spooling: Ipc Ipq Ipr Iprm Iptest pac 

e Print job file format conversion: magicfilter bmp- 
foppm djpeg dvips fig2dev giftopnm gs pngtopnm 
pnmtops rasttopnm sgitopnm tiff2ps 


Conclusions 


Open source operating systems present unique 
opportunities for creation of ultra light systems tuned 
to specific tasks. By exploiting the flexibility and rich 
knowledge base of systems like Linux, FreeBSD, 
OpenBSD, and other open source systems, administra- 
tors can greatly reduce the time required to set up, 
maintain, and upgrade the systems under their control. 
LxA facilitates this process by providing documenta- 
tion, tools, and ready-to-run reference implementa- 
tions for several useful Linux based network appli- 
ances. By taking a different approach to composing its 
Linux based platform, LxA reduces system complex- 
ity while increasing security and reliability. In addi- 
tion, LxA will hopefully be able to quickly incorporate 
new versions of critical system components when 
available due to its use of standard components wher- 
ever possible. When compared to commercial UNIX 
systems, Linux presents more options for the average 
system administrator in building unusual or small sys- 
tem configurations due primarily to the following fea- 
tures: 


passwd tool. As discussed previously, the standard passwd 
utility will not function correctly with a read-only /etc direc- 
tory, and this obviously causes problems for a CD based root 
filesystem. The cryptwd tool (in combination with vi) will al- 
low the changing of user account passwords on a standalone 
LPA-CD system. To use cryptwd, one would first open the 
/etc/passwd file in vi and then use an ex command like the 
following: :r ! cryptwd <newppassword>. 
This will place a string representing the crypted version of 
the password specified into the current editing session. If it 
is necessary to include special characters in the password 
string, they may be escaped with a ‘\’ like this; :r! 
eryptwd \%\&chevy. Of course, this command can 
also be used at a Shell prompt if the crypted version of some 
string is needed. This utility chooses a crypt salt for the 
password randomly, so there is no need to specify one on the 
command line. 
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e A standard and flexible RAM disk device 

e The initial RAM disk boot capability 

e Ease of composing minimal systems suitable 
for running from removable media 

e Generally more sophisticated tracing tools 

e Free access to source code for troubleshooting 
and modifications where necessary 

e An overall system ethos focused on flexibility 
and hackability 


The LxA project has definitely succeeded in pro- 
viding me with a more thorough understanding of the 
components involved in bootstrapping and running 
Linux and other UNIX-like systems. In more practical 
terms, LxA configurations have made it feasible for 
me to continue supporting customers located at great 
geographic distances from my new home without 
excessive travel or expense. As with any open source 
project, it is difficult to estimate accurately any rate of 
deployment or success for LxA outside the realm of 
my own experience. I can say, however, that there 
have been approximately twenty to thirty thousand 
visits to the LxA project site since it was first created 
around December of 1999. Of those who have visited, 
at least several thousand have taken the time to down- 
load one or more of the sample configurations. Of the 
several dozen users who have sent me email regarding 
LxA, all have reported positive results and at least 
several have suggested new features and configura- 
tions which they would find useful. I have received at 
least a dozen replies to a request for usage reports so 
far, with those responding most often finding a use for 
the LRA and LPA-CD configurations. Purely subjec- 
tive observation suggests that a large proportion of 
active LxA users are from outside the U.S., and I 
would suppose that this can be attributed to the 
emphasis the project places on its relatively low 
resource requirements and the prevalence of older 
hardware platforms in many areas of the world. 


Future Work 


TODO List 


e More elegant make scripts and better support 
for RedHat based host distributions. 
e Analysis scipts to automate the process of iden- 
tifying and collecting all the components and 
dependencies of new services for LxA configu- 
rations 
A simple console dialog and/or X based GUI 
configuration utility for building LxA root file 
system trees and disk images. 
Enhancement of the LPA-CD root file system 
image with additional components to form a 
more general-purpose ‘LxA Platform CD’. This 
will include sshd for remote administration as 
well as the netatalk package for support of 
MacOS printing clients and some other services 
such as dhcpd and pppd. 
Integration of the Linux kemel 2.4 when 
released. 
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Alternative Configurations 


e LLA (Linux Lynx Appliance): Useful for turn- 
ing ancient 386 class machines into text based 
POS terminals, etc. 

e LMA (Linux Mozilla Appliance): Another CD 
based configuration; the classic thin client for 
creating web and Java based network applica- 
tion systems. 

e LCDA (Linux CD-ROM Tower Appliance) 

e LTSA (Linux Terminal Server Appliance) 

e LVA (Linux VPN Appliance) 


Availability 


All of the disk images, documentation, and other 
materials related to LxA are publicly available via the 
Internet [12]. All software associated with LxA is cov- 
ered, as are the components on which it is built, by the 
GNU General Public License [6]. Everything docu- 
mented in this paper is freely available for any use as 
long as the distribution requirements of the GPL are 
foll owed. 


Errata 


Any corrections or additions to this paper will be 
posted on the LxA project site [12] where the full text 
of this paper will be posted and maintained n HTML 
format as well. 
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Automating Dual Boot (Linux 
and NT) Installations 


Rajeev Agrawala, Rob Fulmer, & Shaun Erickson — Lucent/Bell-Labs Research 
ABSTRACT 


This paper presents a solution for automating dual boot Linux and Windows NT 4.0 
installations on a PC. The process is automated to the extent that one starts with a PC with a blank 
hard disk and ends up with a fully customized dual-boot PC. The process involves booting a PC 
with a floppy disk; choosing what kin of system 1s required from a menu and then, once the install 
begins, popping out the disk. The unattended PC goes through installing and customizing the 
Linux installation. It then automatically boots into DOS and installs Windows NT 4.0. The whole 
process takes about an hour or two, depending upon the type of installation and network speed. 
After the process is complete, the PC is ready, fully customized to the user’s local environment. 
The solution presented in this paper can be used to do either Linux-only or NT-only installs in 


addition to dual-boot configurations. 


Introduction 


Automating the operating system’s installation 
has always been a topic of interest to System Adminis- 
trators. In the era of installing Windows 3.1 and appli- 
cations, automation meant eliminating the process of 
feeding 30 floppy disks one by one, only to find out 
that the 27th disk had a bad sector error. Solaris pro- 
vides JumpStart, with which one can automate OS 
installation. IRIX, in its latest release (6.5), provides 
Roboinst to automate the OS installation. We have 
automated Windows NT 4.0 installs [1]. Linux (Red- 
hat) provides Kickstart to automate the OS installation 
[2]. 

All of these methods automate the process of one 
OS installation only. Our environment initially con- 
sisted of Unix workstations on desktops. Then the 
trend shifted towards PCs running Windows NT 4.0 
on desktops and Unix servers for the backend process- 
ing. With the popularity of Linux, more and more 
users in our environment started asking for PCs run- 
ning both Linux and NT with a choice to boot in either 
operating system. With this trend, the support group 
had to spend a lot of time on each PC to make it dual 
boot. Also, each installed PC was one of a kind, 
depending upon the way a particular administrator 
decided to customize and install it. The method 
described in this paper was developed so those system 
administrators could install PCs more efficiently and 
consistently. 


Overview 


The installation process uses a modified “‘boot- 
net” disk from RedHat Linux Kickstart. The PC 1s 
booted off this disk, and a menu is presented to the 
administrator. The menu typically consists of the 
choices for different ways of customizing the installa- 
tion. The choices, for example, could be 

e Both Linux and NT install customized for 

department A, or B, or C. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


e Only Linux insta ll for department A, or B, or C 

e Only NT install for department A, or B, or C 

e Generic Linux and NT install (No customiza- 
tion) 

e Generic Linux instal] 

e Generic NT install 


Once the system administrator chooses the 
installation type from the menu, the installation starts. 
At this stage, he can pop the floppy disk out and walk 
away from the PC. From this point onwards, the 
installation goes unattended. When the installation 1s 
finished, the PC is ready with both operating systems 
and is usable. Since Redhat Linux does not support the 
latest video cards out of the box, we skip the installa- 
tion of X altogether for consistency. Also, sound 
installation 1s not supported through kickstart. There- 
fore, with our current setup, one has to configure X 
and audio after the installation 1s done. 


The steps involved in doing a dual boot installa- 
tion are: 

e Starting the first OS installation 

e Customization of first operating system 

e Preparing/repartitioning and formatting disk for 
second OS installation. 

e Copying the second OS installation files to the 
partition prepared in previous step. 

e Handing over control to second operating sys- 
tem install program. 

e Second operating system installation and cus- 
tomization. 

e Passing control back to first operating system. 


The first step above is started manually by the 
administrator. The rest of it happens automatically. 
Constraints 


The seven steps mentioned above impose some 
restrictions on what can be chosen as the first operat- 
ing system and what can be second. 
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The following problems need to be addressed to 
be able to do a successful install: 

1. The installation mechanism should be capable 
of installing either operating system or both. 

2. There is some customization required, based on 
departments, user requirements, etc. Therefore, 
it should be possible to specify the type of cus- 
tomization for both operating systems at the 
start of the installation process. 


Default deptA 
Prompt 1 
Display dualboot.msg 
Fl dualboot.msg 
F2 other.msg 
F3 expert.msg 
F4 param.msg 
F5 rescue.msg 
EG. boot.mse2 
F7 snake.msg 
Label linux 
Kernel vmlinuz 
Append initrd=initrd.img network 
Label text 
Kernel vmlinuz 
Append initrd=initrd.img network text 
Label expert 
Kernel vmlinuz 
Append expert initrd=initrd.img network 
Label ks 
Kernel vmlinuz 
Append ks initrd=initrd.img network 
Label deptA 
Kernel vmlinuz 
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3. The first operating system should have knowl- 


edge about the filesystem for the second operat- 
ing system and should be able to make the file 
system bootable for it. 


We shall see later in this paper how the first two 


requirements are met. The third requirement makes 
Linux a natural choice for the first operating system, 
since Linux can create a DOS filesystem through 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ks.cfg profile™deptA 


Label 1 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ks.cfg profile=deptA 


Label generic 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ks.cfg profile=generic 


Label 2 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ks.cfg profile=generic 


Label ntonly 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ntonlyks.cfg profile=ntonly 


Label <3 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/ntonlyks.cfg profile=ntonly 


Label deptAlinux 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/linux-server.cfg profile=deptAlinux 


Label 4 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/linux-server.cfg profile=deptAlinux 


Label genericlinux 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverlIPAddress:/kickstart/linux-server.cfg profile=genericlinux 


Eabed.> 
Kernel vmlinuz 


Append initrd=initrd.img network ks=nfs:serverIPAddress:/kickstart/mathsummer.cfg profile=genericlinux 


Figure 1: Modified SY SLINUX.CFG file. 
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available tools, and, with some effort, it is possible to 
make the DOS partition bootable. 


The Details 


First we place the Linux distribution on a Unix 
server. This can be downloaded from the RedHat Web 
site at http://www.redhat.com. This distribution should 
be exported read only through NFS. 


Boot Floppy 


We next create a boot floppy from the bootnet 
floppy image provided by the RedHat Linux distribu- 
tion. We then modify the syslinux.cfg and create a file 
named dualboot.msg to include the menu to be pre- 
sented to the administrator. Figure 1 shows part of 
modified syslinux.cfg file for the possible menu 
choices mentioned earlier. Figure 2 shows the dual- 
boot.msg file, which 1s presented to the system admin- 
istrator at startup. 


Let’s analyze the line shown in bold in Figure 1. 
This line is used by syslinux to invoke the kemel, 
when the administrator chooses ‘dual boot install for 
department A’. This line tells the kernel to use kick- 
start to do the Linux installation and use the file ks.cfg 
from serverIPAddress as the kickstart configuration 
file. Here the administrator will need to replace the 
‘“serverIPAddress” with the IP address of the NFS 
Server where the ks.cfg file 1s stored. This line also 
specifies ‘profile=deptA’. This is not a kickstart or 
kemel directive. The kemel is passed this parameter 
profile but ignores it. Later in the postinstall section, 
we retrieve this parameter from the kernel and use it to 
customize Linux and NT. Syslinux specifies two con- 
figuration files. One is the kickstart configuration file, 
used by kickstart. The second is a customization con- 
figuration file, which is used by the configuration con- 
trol program ksconfig. Ksconfig is a perl script, which is 
the heart of the installation system. It takes control of 
the installation during the Y%opost section of kickstart 
and then drives the rest of the installation process 
using the directives in the profile parameter. 


Figure 2 shows the ‘DUALBOOT.MSG’ file. 
This file is displayed when syslinux starts from the 
boot floppy. The text in the file acts as the menu text 
for the administrator. 
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Note that by passing the profile parameter to the 
kemel, we solve the first constraint mentioned in ear- 
lier. As we will see later, the profile specifies the cus- 
tomization for both the operating systems. 


Kickstart 


We encourage the reader to read about kickstart 
in reference [2]. Here we give a brief description of 
kickstart, as it relates to this paper. 


Figure 3 gives a typical ks.cfg file for used for 
Linux installation. Here we provide the anatomy of 
important lines and their significance. For the sake of 
clarity, we have added line numbers, but they are not 
part of the actual ks.cfg file. 


Line 3 instructs kickstart to use network mode 
and use DHCP to get IP information. 


Line 5 specifies /linux/redhat-6.2 as the distribu- 
tion directory using NFS from kickstartServer. 


Line 9 says to restore the standard MBR to the 
disk. This option didn’t work for us for some reason, 
so we had to do it later in the customization section. 


Line 11 instructs kickstart to delete all the parti- 
tions on the disk. 


Line 13 creates a partition of type ext2 which 
will be mounted as /boot. The grow parameter on this 
line causes the partition to grow to a maximum. In 
reality, this will be constrained, as we will see later. 


Line 14 creates a swap partition of size 256 MB. 


Line 15 creates an ext2 type partition, which will 
be mounted as /. The grow parameter on this line 
causes it to grow to the maximum available disk 
space. 


Since Line 13 and Line 15 both specify the grow 
option, both the partitions compete for available disk 
space and get roughly half of the total disk space, 
minus the swap partition. Kickstart also restricts the 
/boot partition to be a maximum of 1024 cylinders 
(due to boot restrictions of LILO), so the /boot parti- 
tion gets a maximum of 8GB disk if the disk 1s larger 
than 16GB. Also, as of the writing of this paper, Kick- 
start will create a primary partition for /boot and put 
swap and / in an extended partition. 


Line 17 sets the kickstart mode to install rather 
than upgrade. 


Welcome to Red Hat Linux 6.1! 


Please choose from the following options 


1. deptA 
2. generic 


(Dual boot install linux customized for deptA) 
(Dual boot. install linux not -cistomized) 


(use deliver script before delivery) 


je) bony 


(Finally only NP as Lert installed ‘on, the disk) 


4. genericlo (only Linux installation ~ uncustomized) 
You can still type commands like ks profile=<someprofile> or 
Use other kickstart config file by specifying in DHCP configuration. 
You can still use other linux boot floppy commands. 
[Fl-Main] [F2-Other] [F6-Linux Boot Screen] 


Figure 2: DUALBOOT.MSG file. 
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Line 23 instructs kickstart to skip the X server 
installation. If we don’t skip this, kickstart will fail for 
an unsupported video card. 


Line 25 sets the root password for the machine. 


Line 29 instructs LILO to install its boot sector 
on the /boot partition instead of in the MBR. 


Line 31 sets up kickstart to reboot automatically, 
instead of waiting for the user to press retum. 


Line 34 specifies installation of the complete 
RedHat distribution. 


In the %post section, Line 38 NFS mounts the 
directory which contains the ksconfig file. 


Line 39 executes the script ksconfig. From this 
point onwards, the ksconfig script takes control of the 


1. lang en_US 
me 


network ==bootproto <dhep 


i 


. Hf 
. keyboard us 
. # 
9. zerombr yes 
10. # 
Lis -clhearpart =a li: 
Apes 
13. part /boot --size 1 -grow 
14. part swap --size 256 


152. (part. / “-<size 1! “grow 
16. # 

A. eS eaelek 

18. # 

19. mouse logimmanps/2 
20%. 

21. timezone US/Eastern 
DD 

23. SkKipx 

24. # 

25°. LOOtDW '#%&wtqr 

26. # 


27. auth --useshadow --enablemd5 


28. # 


29 Tilo =slocation iparta tion 
30. # 

31. reboot 

32. # 


33. *packages 

34. @ Everything. 
35. # 

36. *post 

37... (/bin/mkdir /mnt2 
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installation. The complete functionality of this script is 
discussed later. 


Ksconfig Script 


During the %post section of kickstart, the kscon- 
fig script is executed. Before the system’s first reboot, 
this script installs itself to be run during runlevels 1 
and 3. This script also changes the /etc/inittab file to 
set the default runlevel to 1, so once kickstart is done 
and the machine reboots, 1t goes into runlevel 1. 


The ksconfig script is mostly profile driven. It 
extracts the name of the profile, which was passed to 
the kernel by syslinux at the start of installation. 
Although Linux does not provide any direct way of 
obtaining the invocation time parameters, it keeps 
them in the /proc/cmdline file. The ksconfig script uses 


2 
3 
4 
5. nfs =-<server kickstartServer --dir /linux/redhat-6.2 
6 
7 
8 


38. /bin/mount serverlIPAddress:/kickstart /mnt2 


39. /mnt2/ksconfig 
40. /bin/umount /mnt2 
41. /bin/rmdir /mnt2 


Figure 3: A typical kickstart configuration file (ks.cfg) used for Linux installation. 
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this file to get the profile name. The script assumes 
that all the profiles are available in the directory from 
which it was run. It copies the profile to the local sys- 
tem. A sample profile is given in Figure 4. This profile 
is divided into sections. Each section header is 
enclosed within ‘%’ characters. 


The format of the section header is given below: 
%SectionName runlevel-validpasses% 


SectionName is the name of the section. The valid 
section names are: 
e Environment 
Disklayout 
Nt 
Linux 
Installapplications 
Prepnt 
Config 
Zerombr 
e Removepackages 
e Cleanup 
The section names are case insensitive. 


Runlevel-validpass is an optional control string. 
If present, runlevel is the Linux runlevel, during which 
this section is active. So, with a section header speci- 
fying a runlevel of 3, it will be processed during 


yenvironment% 
logFile=/var/log/ksconfiglog 
dualboot=true 
*endEnvironment% 

*%diskLayout 1-1% 

##Hsaves /boot partition on root 
partrootdevl=saveOnPartrootdev6 
#Hunmounts /boot partition 
partrootdevl=umount 

#Hdeletes /boot partition 
partrootdevl=delete 


##}Creates a 30M ext2 partition starting from cylinder } 


partrootdev1=30M,ext2,1 

##Makes e2fs filesystem 
partrootdevl=mke2fs 

#Marks this partition active 
partrootdevi=markactive 

##fmounts it on /boot 
partrootdevl=mounton /boot 

##Hrestores the original /boot contents 
partrootdevl=restoreFromPartrootdev6 
#tHereate a 2047M DOS Partition after /boot partition 
partrootdev3=2047M, FAT16,afterpartrootdevl 
yendDiskLayouth 

ant 

*%diskLayout 3-1% 

partrootdev3=mkdosfs 

*endDiskLayout% 

*installapplications 3-1% 

acrobat 

audix 

exceed 

gsview 

mcafee 

netscape 

office97 

reskit 
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runlevel 3 only. During the “%post section of kickstart 
(before the machine is rebooted), the runlevel is not 
set. To process a section during the “%post section 
itself, runlevel should be specified as ‘1’ (signifying 
install runlevel). 


Validpass is a combination of the following val- 
ues: 

e 0-9 — A number specifying the installation pass 
during a particular runlevel (e.g., 1 means first 
reboot in the specified runlevel, 2 means 2nd 
reboot in the same run level etc.). 

e ‘d’ — The installation pass is set to ‘d’ when the 
ksconfig script is manually run to customize a 
machine after a generic installation. The letter 
‘d’ signifies delivery pass (when the PC 1s to be 
finally delivered to the user). 

For example, if the administrator decides to process a 
section during runlevel |] and pass 2, he will specify 
1-2 in the section header. When the ksconfig script first 
boots into runlevel |, it sets the pass number to 1. 
Since the section is valid in pass 2, it will reboot the 
machine once again in run level | (after processing 
other applicable sections), increase the pass to 2 and 
then process the section. Specifying ‘1-1d’ as the con- 
trol will execute the section after first reboot in 


security 

ssh 

winzip 
*%endInstallApplications% 
*prepNt 3-1% 
hostname=useDummy 
preparent=rootdev3 
yendPrepNt% 

*endnt% 

*linux% 

*zerombr i-1% 

rootdev 

*endZerombr% 
*removePackages 1-1d% 
rpmRemCmd=/bin/rpm --quiet +-allmatches --nodeps -e 
cxhextris 

ghome- games 
gnome-games-devel 
gnuchess 

howto -chinese 
kdegames 


yendRemovePackagesh 


AONE spe ee 
copyFile=functions 
customFile=deptA.1-1 
*endConfig% 


*%config 3-1d% 
customFile=deptA.3-1 
*endConfigh 

*cleanup 3-1d% 

/save 
/etc/re.d/init.d/ksconfig 
%endLinux% 


Figure 4: A sample profile ‘deptA’ for ksconfig. 
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runlevel 1, and again if the profile is used to manually 
customize a generic machine. 


To better understand the use of pass “d’, consider 
a scenario where PCs are ordered in bulk for various 
departments and kept preinstalled. Since, at the time 
of installation, it is not known which department a par- 
ticular PC will go to, all the PCs are initially installed 
with no customization. At the time of delivery, the 
ksconfig script can be run manually with the profile 
name for the particular department. At that time only 
sections marked with pass ‘d’ will be executed. So the 
same profile can be used either at the time of cus- 
tomized installation from the start or during manual 
customization. 


The %endSectionName™% directive marks the 
end of a section. Sections can be nested. 


Let’s now look at the sample profile ‘deptA’ 
given in Figure 4. The profile is read top to bottom by 
the ksconfig script. Any sections not valid for the cur- 
rent runlevel and pass are skipped. 


Now we shall discuss the various sections. This 
discussion is not necessarily in the same order as it 
appears in the sample profile given, but in the 
sequence in which the sections are executed. 


Environment Section 


The first section name 1s ‘environment’. There is 
no control string, so the section will be executed in all 
run levels and passes. The first command in this sec- 
tion is logFile=filename. Ksconfig will log the installa- 
tion in this file. The only other command in this sec- 
tion is dualboot=true. This tells the ksconfig script that 
this machine is going to be a dual boot system. The 
other choices are ‘linux’ or ‘nt’ 


Zerombr Section 


The Zerombr section is executed in runlevel ‘1’. 
This means it is executed in the “post section of kick- 
start. The only command in this section is ‘rootdev’. 
This command instructs ksconfig to restore the Stan- 
dard MBR to the root disk. For a discussion on what is 
a root disk, see the next section. This section was 
introduced as a workaround to Linux Kickstart’s 
zerombr [2], as the command actually did not restore 
the standard MBR (as of RedHat 6.1). To do this, the 
ksconfig script dd’s (the Unix dd command) 440 bytes 
of standard MBR to /dev/hda (or the root device). 


DiskLayout Section 


In the example profile, there are two DiskLayout 
Sections. The first section is processed in first pass of 
runlevel 1. This means it is done after kickstart is fin- 
ished and the machine reboots the first time in single 
user mode. This section does the disk repartitioning 
for the installation of Windows NT. As you will recall, 
during kickstart, we created two ext2 type partitions, 
roughly the same in size, Occupying half of the disk 
each. The first partition was mounted on /boot, and the 
2nd partition was mounted on ‘/’. Now we will save 
the contents of the /boot partition, delete the /boot 
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partition, create a 30M /boot partition and restore the 
/boot files saved earlier. In the rest of the empty space 
so created, we will make a 2047MB, FAT16 partition 
and make a DOS filesystem on it. For a dual boot 
installation, this assumes a minimum disk of size 5 
GB. This value can be tweaked, based on the standard 
NT load. However, if the disk is larger in size and this 
partition is greater than 2GB, the rest of the partition 1s 
not wasted. The NT installation grows this partition to 
the available size. Figure 5 shows the disk partition- 
ing as the installation proceeds. 


The First command in the ‘diskLayout’ section 

1s: 
Partrootdevl = saveOnPartRootdev6 

This command instructs the ksconfig script to find out 
the base root device. This would be hda in normal IDE 
based systems, sda in SCSI based systems and hde if 
Ultra66 IDE card is used. The ksconfig script takes par- 
tition 1 of the root device and saves it as a tar file on 
partition 6 of the root device. As can be seen in Figure 
5(b), partition | is the /boot partition and partition 6 is 
where ‘’/’ is mounted 


The rest of the commands in this section: 

e Unmount the /boot partition. 

e Delete the partition. 

e Create a 30MB ext2 partition. 

e Make an e2fs filesystem on this partiton. 

e Mark the partition active. 

e Mount the partition on /boot 

e Restore the partition contents saved earlier. 

e Create a 2047MB, FAT16 partition after the 
/boot partition. 


The second DiskLayout section is executed in 
runlevel 3 and is ignored in runlevel 1. This section 
instructs ksconfig to create a DOS filesystem on parti- 
tion 3. Partition 3 was already created as type FAT16, 
in the first DiskLayout section. 


Removepackages Section 


This section is executed in runlevel 1 and passes 
1 and ‘d’. This means that if this profile is used during 
the automatic dual boot install process, it will be pro- 
cessed the first time the machine boots in runlevel 1. 
Also, if this profile is used to manually customize a 
generic Linux installed system, this section will be 
executed. This section lists various RPMs which 
should be removed from the system. The ksconfig 
script reads one package name at a time and removes 
the package. By default ‘rpm -quiet -e’ is used to 
remove the package. If other packages have dependen- 
cies on the current package, it will not be removed. 
One can specify a command to remove the package by 
the directive rpmRemoveCommand=command. This 
way, if the administrator wants, he can remove the 
package regardless of dependencies. 


Config Section 


This section facilitates the execution of any 
script/program to customize the system. In the 
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example profile, there are two Config sections. The 
first section 1s processed in runlevel 1 pass 1. The 
valid commands in this section are copyFile and cus- 
tomFile. The CopyFile command simply copies 
the file/directory from the NFS mounted directory to 
the jocal disk on the system. The CustomFile com- 
mand copies the filename specified to local disk and 
executes it. Though the command execution is done in 
the runlevel/pass specified in the header, the actual 
copy operation is done during the “post section of the 
install itself. If networking is not available (as in sin- 
gle user mode), the commands can still be executed. 
This section is an exception, in the sense that ksconfig 
script does not completely ignore this section during 
the %post phase of the installation. For security rea- 
sons, one would like to execute a configuration script 


a) Empty Disk 


- Primary 
Partiti 


Extended 
artition 
hda2 





le? 


c) Disk partition for NT 
install 
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to set the root password and turn off certain services in 
runlevel 1, and then do the rest of the customization in 
runlevel 3 when all the networking is available, such 
as NIS, mounts etc. 


InstallApplications Section 


This section lists the applications that should be 
installed on NT. During this section the ksconfig script 
only makes a list of applications to be installed. This 
list is used during the prepNT section (discussed 
later), to generate a bat file (apps.bat) for NT to install 
these applications. 


PrepNT Section 


This section prepares for NT installation on the 
DOS partition created earlier. The only valid com- 
mands in this section are hostname and prepareNT. If 


a 


Primary 
] 


artiti 
hda 


Extended 
artition 
hda2 





1+? 


b) Disk partition by linux 
install 


Primary 
Partiti 


ale 


Extended 
artition 
hda3 





l_—? 


d) Disk partition after NT 
install 


Figure 5: Disk partition during installation process. 
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the hostname is specified to be ‘useDummy’, the 
script generates a dummy hostname and assigns it to 
the NT side. This is required because NT 4.0 does not 
use the hostname provided by DHCP. One of the items 
in my TODO list is to provide a better solution for set- 
ting the NT hostname. 


The second command, prepareNT, specifies the 
partition that should be prepared for NT installation. A 
lot goes on behind the scenes to prepare the partition. 
A DOS filesystem was already created on the partition 
during the DiskLayout section. 


Our automated procedure for loading NT relies 
on having a DOS partition pre-loaded with certain 
programs and data files. This is later converted into an 
NTFS partition. We use Linux tools to create and pop- 
ulate this partition. The entire process is described 
below: 


Making Disk DOS Bootable 


The first thing we need to do is to make it DOS 
bootable. For this, we will copy the necessary DOS 
files 10.sys, msdos.sys and command.com to the parti- 
tion. We will also copy the DOS boot sector to the first 
512 bytes of the partition. 


Setting Up Networking 


Within Linux, we know about the Ethemet 
adapter, so we generate the necessary LANMAN con- 
figuration files and copy them to the DOS partition 
along with other LANMAN driver files for the Ether- 
net adapter. We keep all these files in a subdirectory 
under the same directory where ks.cfg (the kickstart 
configuration file) is kept. 


Setting up to start installation 


We next copy the config.sys and autoexec.bat 
files, which set up the MS LANMAN networking, 
map a directory from the network which contains the 
NT installation and other application installation files, 
and then runs the NT setup program to start NT instal- 
lation. We also generate a file called apps.bat, which 
contains commands to install applications mentioned 
in ‘installApplications’ section (described earlier). 


Passing control to DOS 


After copying all the necessary files to the DOS 
file system, we need to pass control to DOS, to start 
NT installation. We do this by modifying the 
LILO.CONF file. LILO is set to boot into DOS by 
default and then the system is rebooted. 


Once control is passed to DOS, the NT installa- 
tion starts. The rest of the NT installation and cus- 
tomization process is described in detail in Autoinstall 
for NT [1]. 

Getting Control Back to Linux 

During the installation, NT makes its own parti- 
tion active. Therefore when the NT installation fin- 
ishes, the system immediately boots NT, without 
going through LILO. We run a script as the last part of 
the NT installation to mark partition 1 active again. 
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This makes the boot process go through LILO, giving 
the choice of booting into either NT or Linux, with NT 
being de fault. 


Cleanup Section 


The cleanup section removes the files that were 
copied onto the local disk during the installation pro- 
cess. 


Installing Other Combinations of Operating Sys- 
tems 


We discussed the dual boot installation process 
above. Along similar lines, other combinations of 
OSes can be installed. The SYSLINUX.CFG file in 
Figure 1 shows the commands to invoke kickstart for 
other combinations of OSes. Notice that the configura- 
tion files for kickstart and ksconfig are different for 
each combination of OS. For example, to start a 
Linux-only system, we will use a kickstart file which 
will only have 30 MB for boot partitions to start with, 
instead of using the grow option. Also in the ksconfig 
profile, we will remove all the NT-related sections 
(DiskLayout & PrepNT etc.). 


To install an NT-only system, we change the 
Kickstart configuration file to do a minimal Linux 
installation instead of everything (enough for our pro- 
cedures to work), and in the disk layout section, we 
don’t create a 30MB /boot partition. Instead, we create 
a DOS partition starting at cylinder 1. We also delete 
the extended partition (containing the Linux root and 
swap partitions) before rebooting to go into NT. 


Linux and Device Drivers 


There are some situations where Linux drivers 
for the PC hardware being installed are not available. 
If the drivers are not critical to the Linux boot process, 
then the installation will complete without any prob- 
lem. For example, if Linux does not support the Video 
card or Sound card in the system, this will not affect 
the installation. However if Linux does not support the 
IDE controller (for the hard disk), such as an Ultra 
66/100 IDE card, then the default RedHat installation 
will not work. This problem can be solved if there is a 
kemel patch available for the device in question. 
Using the patch, a new kermel can be compiled for the 
boot floppy. Also, new RPMs should be created for 
the updated kernel and should be added to the Linux 
distribution. Now the installation can be done on the 
system in the usual manner. Once you have created a 
new distribution and boot floppy, you can continue to 
use this for PCs which may not have the new hard- 
ware in them. This means that you only have to main- 
tain one Linux distribution and boot floppy for 
installing systems. As you add support for new hard- 
ware, you are updating the distribution and floppy, 
which you can use on all systems. Only having to 
maintain a single boot floppy for many kinds of instal- 
lations has turned out to be a good thing. In particular 
it reduces complexity, reduces the work involved in 
maintaining the system, and reduces confusion. 
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Alternative Install Techniques 


Dual boot installation can be automated using 
other techniques, in addition to what is described in 
this paper. Those techniques include 

e Cloning the hard disk. 
e Copying the NT image during the “post sec- 
tion of kickstart. 


In the first method both the operating systems 
and applications are installed on a “‘standard PC”’ in 
the desired manner. Once the install is done, the disk 
is kept as a master copy for the install. For any new 
PC install, a simple disk copy is done onto new PC 
disk and after host specific customizations installation 
is complete. 


In the second method, kickstart is done pretty 
much the same way as described in this paper, but dur- 
ing the Y%opost section, instead of installing NT, a pre- 
viously made NT image is copied to the disk. 


Both of these methods provide very good turn- 
around time for installing PCs in bulk. 


However both of these methods have some dis- 
advantages that made us decide to develop our current 
method. If there are many different hardware configu- 
rations, you may need an image per hardware configu- 
ration. If there are different types of software configu- 
rations required, again, an image per software configu- 
ration may be required. If there are many combina- 
tions of both software and hardware configurations, 
then the number of images required can multiply 
rapidly. If a change is needed, a patch or new soft- 
ware package or whatever, many of these images may 
require changes. The amount of work required to 
maintain this seemed too great to be worthwhile. In 
the case of true installation, one has to upgrade the 
operating system or application in the distribution 
only. The next time a new PC is installed, it automati- 
cally gets the upgraded OS or application. 


The second issue with cloning or copying NT is 
that all the cloned systems will share the same SID. 
When these systems are put on the network, Local 
Administrator on one system will be treated as 
Administrator on all the cloned systems, because they 
share same SID, even though they may have different 
passwords for administrators. This is true for both 
Administrator and non-Administrator accounts. There 
is software available which can reset the SID to some 
random generated value after cloning the disk (like 
Norton Ghost [3]), but Microsoft does not support 
resetting the SID. 

The advantage of using the method described in 
this paper is that both operating systems are installed 
using their native install procedures on the PC where 
they are going to be used. 


Evaluation of the Technique 


We have been using this technique to do installs 
for more than 6 months (as of the writing of this 
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paper). Our PC install group is very satisfied with this 
method. We have not faced any major problems 
although we have found some unsupported hardware 
in Linux. We solve this problem by recompiling the 
boot kernel and upgrading the kernel in the distribu- 
tion. We have been requesting the Linux Kickstart 
developer community to support a “pre section in the 
kickstart configuration file, similar to the %post sec- 
tion, with the difference that commands specified in 
Yopre section would be executed before Linux installa- 
tion begins. In that case we would be able to partition 
the disk the way we need it from the start, and would 
not have to jump through hoops to repartition the disk. 
It appears that a %pre section may be available in a 
future release of RedHat. 


To Do List 


Though the system described above works quite 
nicely, there is room for improvement. Some of the 
major items in our TODO list are mentioned below: 

e Solving the NT hostname problem described 
earlier. 

e Automatically installing/configuring X and the 
sound card during Linux installation. 

e A more ambitious item is to extend the system 
to install any two (or possibly more) operating 
systems (especially to support Linux and Win- 
dows 2000 dual-boot systems). 


Source 


The source and complete profiles to do different 
combinations of operating systems (NT and RedHat 
Linux) are available from the author by sending email 
to dualbootinfo@research.bell-labs.com. 
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ABSTRACT 


We describe a system to record and analyze “raw” Frame Relay and point-to-point T-1 
packets. The data are captured by “eavesdropping” on the HDLC transmit and receive lines 
between the router and CSU/DSU. Analysis of the data provides circuit and application utilization 
information on a one-second or shorter time scale. Routine and custom reports are accessible 
through Web interfaces to provide easy access by our global systems and network staff. The packet 
data can also be used to debug applications in the same way as conventional packet capture 


systems. 


Introduction — Why We Needed This System 


Frame Relay networks provide organizations 
with a flexible and economical method of intercon- 
necting sites over a wide range of distances. A major 
source of the flexibility comes from the ability to con- 
nect many circuits over a single access line, such as a 
T-1 (1.5 Mbps) or E-1 (2Mbps, used in Europe). Each 
circuit, called a PVC (Permanent Virtual Circuit), has 
a guaranteed bandwidth, known as CIR (Committed 
Information Rate). Most Frame Relay carriers allow 
PVCs to “burst above CIR’’, possibly to the full band- 
width of the access line. The sum of the instantaneous 
bandwidth for all PVCs can not, of course, exceed the 
bandwidth of the access line. This leads to interesting 
traffic managemient questions. 


Complex Frame Relay networks are often laid 
out in a “‘hub and spoke” arrangement. Multiple hubs 
may connect subsidiary offices in a geographical area. 
The hubs are then joined together, usually with higher 
bandwidth interconnections. 


While debugging Frame Relay network prob- 
lems, for both bandwidth management and application 
issues, we have used tcpdump [McCa97] to record 
packets at the Ethernet interface of routers. We often 
wished, however, that we could see exactly what data 
were flowing in and out of the T-1/E-1 serial access 
lines. This was especially true at Frame Relay hub 
sites where many packets pass through the router, but 
never appear on the Ethernet side because they are 
destined for another site on our network. In addition, 
useful Frame Relay header information is lost once the 
frames are converted to Ethernet packets. 


As we did more application debugging and traf- 
fic analysis it became clear that we needed a system to 
record raw frames outside the router, directly from the 
communications lines. Then we could examine any of 
the Frame Relay header information and as much of 
the data, including IP header and payload, as we cared 
to record. 


Commercial systems were reviewed but none 
were found that met the requirement to record raw 
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Frame Relay packets for more than a few minutes. 
Our company already used two of the more popular 
brands of ““WAN Probes”, but they are mostly useful 
for real-time diagnostics, and RMON (Remote Net- 
work Monitoring Management Information Base) type 
historical data. We considered using Network Flight 
Recorder [Ranu97], but at the time, it could not record 
data from WAN communications lines. 


While most routers count the Frame Relay con- 
gestion notification bits (FECN and BECN, Forward 
and Backward Explicit Congestion Notification) in the 
header, they do not count discard eligible (DE) bits. 
The five-minute counts of FECNs and BECNSs that we 
record via SNMP do not provide any method to assign 
the occurrence to a particular second, or to particular 
packets. Debugging an application / network interac- 
tion problem without the raw packet data is very diffi- 
cult. 


In this paper we will first review the hardware 
requirements and packet acquisition software. Then 
the traffic analysis software will be discussed, fol- 
lowed by real-world analysis examples including 
“Congestion and Circuit Capacity Planning”, ‘‘Using 
the Raw Packet Data’, and “Application Profiling”. 
Two short sections describe the extension of the sys- 
tem for T-1 point-to-point circuits, and using tcpdump 
to perform similar analysis when the packets of inter- 
est are available on a LAN. We will close with some 
ideas for future applications. 


The Hardware 


The system is built on a low cost desktop plat- 
form running RedHat Linux (version 5.2 or 6.x). The 
heart of the hardware is one or more communications 
boards from Sangoma Technologies Inc. (Markham, 
Ontario, Canada). The first few monitors we built used 
two Sangoma WANPIPE S508 ISA cards, but we are 
now using a single Sangoma WANPIPE S5142 PCI 
card that can handle four communications lines at up 
to 4Mbps in “‘listen-only”’ mode. 


Acquiring the bi-directional data requires the use 
of two receive lines and the associated clock signals 
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on the Sangoma cards. The transmit lines of the card 
are not connected. While we have successfully con- 
nected to T-1/E-1 lines using short cables directly 
attached to the communications lines, the use of an 
active ‘‘Multi-Interface Tap” 1s recommended. These 
taps present a high-impedance to the signal lines and 
allow a long cable to be safely used between the tap 
and the computer. The cost for a system to monitor a 
single T-1/E-1, including PC, communications board, 
and WAN tap is about US$2000. A second T-1/E-1 
can be monitored on the same PC for an additional 
US$800. 


Acquisition and Analysis 


The basic model for the system is to record pack- 
ets in both directions (in-bound and out-bound) for fif- 
teen-minute periods. At the end of each period the 
packet files are closed and a new pair of acquisition 
processes are started. Then a summary program pro- 
cesses the data from the previous period. 


This model provides considerable simplification 
at the cost of only about a fifteen-minute delay com- 
pared to a real time system. It also allows convenient 
packaging of the summary results and a method to 
locate raw packet data when deeper analysis is needed. 
The hardware requirements are lower for this post- 
process model; all that is necessary is for the summa- 
rization process to complete in less than fifteen min- 
utes while handling the input streams without packet 
loss. 


Acquisition Software 


The software consists of drivers provided by 
Sangoma, a modified version of Sangoma’s example 
C program for data capture, and a set of Perl programs 
to control the acquisition and analyze the acquired 
data. 


Sangoma’s driver software was set up for 
CHDLC (Cisco HDLC) mode. The packet capture 
program, frpcap, writes files in a format closely mod- 
eled after the tcpdump format. The only significant 
difference is that the Frame Relay header is recorded 
in place of the Ethernet header. The first 150 bytes of 
each raw packet are usually saved to provide context 
information during application analysis. 


The packet acquisition process is driven by 
frcap_run, a Perl script that runs every fifteen minutes. 
Frcap_run stops the current frpcap processes (one for 
each data direction) and immediately starts another 
pair. A traffic summary program, fr_decode, is then 
run on the two packet files. Following the summariza- 
tion the raw packet files are compressed, and any files 
that are older than a preset age are deleted to conserve 
disk space. For a fairly busy set of Frame Relay cir- 
cuits on a T-1 access line, eight days worth of raw 
packet files consumes 3-6GB of disk space. 
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Analysis Software 


The fifteen-minute raw packet files are summa- 
rized by a Perl program that appends its output to a 
daily summary file in XML format. The XML file is 
read by other programs for display and further analy- 
sis. An example of a fifteen-minute summary output 
displayed by a Web application that formats the XML 
data is shown in Figure 1. To reduce the size of Figure 
1, data for a minor PVC were removed and only the 
top five numbers, rather than our usual ten, are shown 
in each category. 


The report consists of five major sections. The 
first section is a PVC summary showing the DLCI 
(circuit number), number of packets and bytes, per- 
centage of bytes per circuit relative to all circuits on 
the access line, congestion notification counts, DE 
(discard eligible) counts and TCP re-transmission 
counts. Since this router does not set any congestion 
notification information (Frame Relay switches further 
down stream set these) or DE bits, the counts are all 
zero for the out-bound direction. Some of our routers 
do set DE for Internet traffic to give it a lower priority 
(setting DE tells the carrier that the traffic can be 
dropped if the network is congested, in exchange the 
carrier does not count the packets towards certain 
credit limits). The TCP re-transmit counts are done 
separately for packets with and without DE so that we 
can determine the effect on packet loss within the 
Frame Relay network when DE is set. 


Layer 2 and 3 protocol counts are summarized in 
the second section. For each protocol observed the 
Ethernet/802.2 type field, IP type by number (if an IP 
protocol), protocol name, number of packets and 
bytes, and percent utilization by bytes for the PVC are 
shown. For TCP/IP a count of packet re-transmissions 
is displayed. 

In the third section we display the busiest and 
quietest seconds for the access line and for each PVC. 
The generation and use of these data are described 
later under Congestion and Circuit Capacity Planning. 


The top sources and destinations of data are 
shown in the fourth section. The protocol, IP address, 
port number, number of bytes, and percentage of the 
total for all traffic on the access line are included. If 
the server name is known it is also displayed, along 
with a short description of the application. 


The fifth section of the report lists the utilization 
of the access line by application. Where possible we 
identify applications by the IP address of the server. 
Although it might make sense to further define appli- 
cations by port number, many of our servers run a sin- 
gle application. In fact we often have two servers run- 
ning the same application in which case packets with 
either IP address will count towards the application. 
Database servers often run multiple applications 
which all use the same IP address/port number pair so 
the addition of a port number qualification would still 
not uniquely identify all applications. In the future we 
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Frame Relay Traffic Summary (Philadelphia) 
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Figure 1b: Frame relay traffic summary for a single T-1 access line (cont. ) 


hope to encourage the use of virtual IP addresses 
assigned on a per application basis to provide a simple 
accounting method. The “New TCP Sessions”’ col- 
umn indicates how many sessions were initiated dur- 
ing the fifteen-minute period and “Total TCP Ses- 
sions” counts both new and ongoing sessions. For 
Web based applications these counts are not very use- 
ful except as a type of hit counter, but for applications 
with persistent connections it is a measure of the num- 
ber of users connected during the period. 


The final two items in the report are a count of 
initial TCP SYNs, which might be used for intrusion 
detection, and a count of TCP re-transmissions for all 
PVCs. 


Figure 2 shows the top of a report for the in 
bound direction. Note that we have a variety of FECN, 
BECN, and DE information related to packets flowing 
in this direction. The FECNs and BECNs provide 
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information on the level of congestion in the carmier’s 
Frame Relay network [Blac95]. The rest of the report 
contains the same information as shown in Figure 1. 


Several customizable reports using the fifteen- 
minute summary data are available. A Web form can 
be used to select a single application and then generate 
a list of “Application Summary” entries for just that 
application. This list of application usage 1s very help- 
ful for determining the bandwidth impact of an appli- 
cation and the number of users. Since we are often 
told “‘There will be 400 users in Europe accessing this 
Philadelphia based application”, we can use the tool 
to judge how many are logged-in simultaneously and 
to monitor usage growth. Another program will gen- 
erate a daily or monthly “Application Summary” by 
summing usage for each application over time. 


The above example data is for one of our busier 
access lines, but it is far from the most complex. One 
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European Frame Relay hub site has 13 PVCs on a sin- 
gle access line. Some of the traffic is business critical 
telnet traffic between subsidiary sites and an AS/400 
at the hub site. Much of the traffic, however, is 
intranet mail or Internet data that comes in on one 
PVC and then heads towards the US on another PVC. 
Without the Frame Relay monitor it would be very 
difficult to determine what applications and protocols 
consume the PVC and access line bandwidth. In some 
cases it is necessary to obtain application summaries 
for a single PVC to determine what is happening on a 
particular circuit. We do not routinely report applica- 
tion and protocol usage on a per PVC basis in order to 
keep the size and complexity of the reports reasonable. 


Congestion and Circuit Capacity Planning 


Like many organizations, we collect router statis- 
tics via SNMP every five minutes. While five-minute 
averages are a useful measure of how a circuit is doing 
in relationship to its bandwidth limit, they do not tell a 
lot about the instantaneous (one second, or smaller, 
time scale) state of a circuit that governs interactive 
performance. If a circuit is saturated for several ten- 
second bursts during a single five-minute period the 
average utilization might appear to be quite reason- 
able. An interactive user, however, would likely say 
that the network was slow while his packets were 
waiting in a router buffer. 


Our monitor attempts to measure the largest 
peaks in a fifteen-minute period by summing the num- 
ber of bytes transmitted and received for each one-sec- 
ond interval. The summary program reports the ten 
busiest seconds for the access line and each circuit. A 
sample plot of busy seconds for one day is shown in 
Figure 3. The presence of many long vertical lines 
connecting the data points is good because it indicates 
a large variation in the top ten busy seconds and there- 
fore there are fewer than ten very congested seconds 
in a fifteen-minute period. The lack of long vertical 
lines around 13:30 GMT indicates significant on- 
going congestion. Quiet seconds are measured in a 
similar fashion. If a usually busy circuit has seconds 
with zero, or a low number of bytes, then it may indi- 
cate a circuit or routing problem. 


We had one problem where a large number of 
quiet seconds were showing up with low utilization 
during busy times of the day. Since there were also 
user complaints, a detailed analysis of the packet data 
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was performed. It showed all IP traffic periodically 
stopping for about eight seconds while IPX was still 
flowing normally. We traced the root cause to an IP 
routing problem. Without the IPX traffic the problem 
would have been easy to spot since there would have 
been periods of about eight seconds with zero traffic 
on an otherwise busy circuit. 


While we have not yet developed a formal rule 
set, it should be possible to determine how well PVCs 
and access lines are sized with respect to the actual 
traffic using busy second data. Clearly, looking at the 
busiest seconds on a circuit is much more meaningful 
than five-minute average data when mission critical 
interactive applications are the most important traffic. 


Using the Raw Packet Data 


Since the raw packet data are stored in separate 
files for in-bound and out-bound directions the two 
files must be combined for traditional packet trace 
analysis. A utility program performs this task by 
putting the packets from a pair of files into a time 
ordered sequence and writing a tcpdump format file. A 
“Frame Relay information” file containing the Frame 
Relay header information is also written. 


We have a packet trace analysis program origi- 
nally written for tcpdump files that can optionally read 
the “Frame Relay information”’ file and list the DLCI 
(circuit ID), FECN, BECN, and DE bits for each 
packet. Using this feature, we have discovered that 
some applications were operating over asymmetric 
routes. In addition to our own analysis programs, the 
tcpdump format files can be examined using other 
programs such as tcpdump itself, or ethereal 
[Ether00]. Since ethereal, and its companion program 
editcap, can export packet data to other formats, the 
traces can be analyzed with popular commercial prod- 
ucts. When a session needs to be followed through 
multiple fifteen-minute periods we use a simple pro- 
gram to concatenate multiple tcpdump files. 


In a previous paper [Meek98] we discussed inter- 
esting issues we have had with our telecommunica- 
tions vendors. Our complaints about a slow circuit 
sometimes yield a vendor response like: ‘‘Customer is 
exceeding CIR by 160%” with the implication that the 
over utilization of a circuit (bursting) has lasted for an 
excessively long period. With raw packet information 
it should be possible to compute bandwidth utilization 
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Figure 2: Portion of in-bound frame relay traffic summary for a single T-1 access line. 
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Figure 3: Busy seconds on a frame relay circuit. This circuit has a CIR of 1024 kbps on a T-1 access line. 
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Figure 4: Five-minute average traffic for the same day and circuit as Figure 3. This standard method of measuring 
utilization hides many significant traffic spikes that are observed in the busy seconds plot. 
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Instantaneous Bandwidth & Congestion Control 
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Figure 5: Instantaneous bandwidth utilization (one-second time scale) and Frame Relay network congestion control 
information (percent of incoming packets with BECN set). 
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on a per-second basis and use the same algorithm as 
the telecommunications equipment to verify when, 
and by how much, CIR was exceeded. Figure 5. shows 
the per-second bandwidth utilization of a circuit for a 
thirty-minute period and the percentage of in-bound 
packets with the BECN bit set to indicate congestion 
on the out-bound circuit. The BECNSs are sent by the 
carrier’s switches to indicate that there is congestion in 
our out-bound direction and that our bandwidth usage 
will be throttled 1f we are exceeding CIR and are out 
of credits. In the future we hope to use this data to 
accurately determine when we truly exceed our con- 
tracted bandwidth and how we might implement qual- 
ity of service to prioritize traffic to manage bandwidth 
bursts. 


Packet loss is an important parameter in any data 
network. One measure of packet loss is the percentage 
of TCP packets that must be re-transmitted. Figure 6. 
illustrates TCP re-transmission rates on one of our 
busy circuits over several months. Hours with fewer 
than 10,000 TCP packets are not shown. Since this cir- 
cuit feeds multiple downstream Frame Relay circuits, 
and many LAN segments, packet loss could occur in 
several places. During late July we had problems with 
a T- 1 interface that caused a significant portion of the 
re-transmissions during that period. Further analysis of 
the raw packet data can determine what destination IP 
addresses were responsible for the re-transmissions. 
We recently added a new section to the standard report 
(Figure 1) that lists the top ten destinations of re-trans- 
mitted packets to help identify hosts or subnets with 
problems. Re-transmitted packets are counted by 
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tracking the TCP sequence numbers by session for 
packets with a payload (not acknowledgement-only 
packets). 


Application Profiling 


Profiling the bandwidth requirements of an 
application is a useful exercise to perform during the 
development or evaluation of a software product. We 
have often found that applications originally devel- 
oped for use on a LAN have serious problems when 
they are moved to a WAN or Intemet environment. 
Problems result from large quantities of data being 
sent, application level handshaking resulting in exces- 
sive round-trip-time waiting, or even the same set of 
data being requested (and delivered) multiple times 
due to a programming error. These issues have 
occurred in both internally developed and purchased 
commercial software. 


One advantage to using the WAN packet capture 
system for application profiling is that the application 
under test can be observed along with all of the other 
data flowing on the circuit at the same time. Since we 
routinely capture all of the traffic on monitored cir- 
cults, no advance preparation is required for most 
application profiling tests. We have found, however, 
that it is helpful when testing an interactive applica- 
tion if the test procedure has timed pauses of 15 to 30 
seconds where the user does not touch their hardware. 
The pauses are used to separate phases of the applica- 
tion in the packet traces and to determine if the client 
and server “‘chatter”’ when idle. 
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Figure 7: IP packet data for a single session extracted from a series of Frame Relay packet capture files. 
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A disadvantage to using the WAN packet capture 
system for these tests is that we generally capture only 
150 bytes of combined header and payload. While this 
small number of payload bytes is often enough to 
determine the context of the application (especially 
when ASCII or EBCDIC data are involved), it might 
not be enough to confidently determine that multiple 
packets with identical payload are being transmitted 
within a session (determined by computing MD5 
checksums on payload content [Meek 98]). 


An example of application profiling is summa- 
rized graphically in Figure 7. The application is a 
widely used ERP (Enterprise Resource Management) 
system that uses page-based terminals as the user 
interface. The user interacts with the system by filling 
out text forms and then transmitting the screen page to 
the server. This method, similar to modern Web appli- 
cations, is an efficient way to implement interactive 
applications on a WAN since reasonable size chunks 
of data, in comparison to keystrokes, are transmitted at 
one time. 


To determine how much bandwidth a single user 
consumes we look at the packets for individual ses- 
sions as a function of time. The top plot shows the dis- 
tribution of packet sizes for each direction, while the 
bottom plot shows the bandwidth used per second. A 
detailed look at the numbers shows that 143,473 bytes 
in 400 packets were sent from client to server and 
112,560 bytes in 348 packets were sent from server to 
the client. By closely inspecting the data using an 
interactive data analysis tool [Grace00] we find that a 
typical transaction is completed in one or two seconds. 
The maximum bandwidth used was 43kbps from 
client to server and 22kbps in the other direction. 


The packet data in this analysis were extracted 
from 30 minutes of raw packet capture data from a 
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Frame Relay access circuit with 13 PVCs. The first 
step was to combine in-bound and out-bound packet 
streams into single tcpdump format files. The tcpdump 
files for the two 15-minute time periods required to 
span the session were then concatenated. Finally, the 
combined file was processed with tcpdump acting as a 
pre-filter to select the session of interest and feed it to 
our own software that summarized the session and 
prepared the packet and bandwidth data for plotting. 


While the tools presented here do a complete job 
of quantifying the actual bandwidth used by an appli- 
cation on a one-second time scale, but they do not 
address simulated scaling of the application. Presum- 
ably, our data could be used as input to a network sim- 
ulation tool to perform the scaling simulation. Our 
tools can, however, select any slice of actual recorded 
network traffic based on source, destination, port num- 
ber, etc. and determine the total bandwidth utilization 
for the applications encompassed by the slice. 


In order to determine how the use of an applica- 
tion changes over time we can look at some parameter 
representing usage. In Figure 8 we show the number 
of sessions per week for one application. The use of 
this particular application varies depending on busi- 
ness cycles and holiday schedules. Other parameters 
useful as a measure of application usage are bytes 
transferred or number of packets. Bytes or packets are 
especially useful for Web, or other non-session or1- 
ented applications. 


Extension to T-1 Point-to-Point Circuits 


We were pleasantly surprised to discover that the 
hardware and software can be used without modifica- 
tion on T-1 point-to-point circuits. The packets on the 
point-to-point circuits have a header very similar to a 
Frame Relay header. The first two bytes of the Frame 
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Figure 8: Usage pattern of a single application by measuring the number of sessions per week. 
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Relay header contain packed data representing the 
DLCI number and the FECN, BECN, and DE bits. 
[Blac95] The first byte of a point-to-point serial line 
packet is set to OxOF for unicast packets and Ox8F for 
“broadcast packets” and the second byte is always 
zero. [Cisc00] For both Frame Relay and point-to- 
point serial the third byte contains the Ethemet proto- 
col code. Note that these conventions may be specific 
to certain types of equipment. 


Related Techniques 


When all traffic of interest is accessible from the 
LAN, simpler tools and techniques should be used to 
record traffic. We use a simple script to start tcpdump 
and rotate the packet capture files on a time schedule 
controlled by cron (the user-mode command sched- 
uler). The acquired data can be analyzed using the 
methods described here for WAN traffic. 


Future Work 


Because we are now moving some of our major 
circuits to ATM in order to overcome T-1/E-1 and 
Frame Relay bandwidth limitations we hope to be able 
to extend the techniques discussed here to ATM. This 
should be straightforward if the ATM interface card 
performs the re-assembly of ATM cells into complete 
IP packets. 


Some of the parameters measured by the system 
will probably be used to generate alarms when they 
exceed certain thresholds. The TCP re-transmission 
rate and quiet/busy seconds are likely candidates for 
alarms. Frame Relay provides information about cir- 
cuits using LMI (Local Management Interface) pack- 
ets. Currently we do not decode these, but plan to add 
the capability in the future. 


We would like to measure the effectiveness of 
QoS (Quality-of-Service) schemes by measuring 
packet delays through the router for packets in differ- 
ent classes of service. This would likely be done using 
tcpdump on the Ethernet side and WAN packet cap- 
ture on the serial line side of the router. Histograms of 
packet delays during busy periods should show that 
the high-priority traffic passes through the router more 
quickly than lower priority traffic. The results might 
be used to tune QoS parameters. 


Conclusion 


We have assembled a low cost Frame Relay and 
T-1 packet capture system with a large memory and 
applied it to real problems. The use of this monitor is 
helping us communicate to management how the 
WAN 1s being used at the application level. It also pro- 
vides detailed worst-case traffic information through 
the analysis of busy and quiet seconds. We have 
upgraded bandwidth on some circuits following analy- 
sis of peak utilization data, identified unusual routing 
problems, and profiled the network impact of applica- 
tions. The large memory and long retention time for 
the raw packet data allow us to troubleshoot many 
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network problems days after they occurred, an impor- 
tant factor in our large, global organization. The data 
analysis discussed in this paper just touches on possi- 
ble uses of raw packet data from WAN circuits. In the 
future, we expect to mine the information in new 
ways. 


Availability 


Supplemental information and some of the soft- 
ware used in the work described here can be obtained 
at http://wanpcap.sourceforge.net. 
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ABSTRACT 


When configuring an IP network, changes may need to be made to several devices as part of 
one configuration operation. Given an in-band access to the devices where data and control 
packets flow on the same network, it becomes imperative to sequence the changes intelligently to 
retain connectivity to the devices that need to be reconfigured. In this paper, we present techniques 
to sequence configuration operations for IP networks. We describe our experiences with 
implementation of the techniques, and experimentally evaluate the usefulness of the proposed 
techniques. We demonstrate that automatic sequencing is valuable in IP network configuration. 


Introduction 


A basic operation performed during network 
operations and management is the configuration of the 
network. Many configuration and reconfiguration 
Operations are typically performed from a remote 
management station. To maintain consistency, several 
devices may need to be updated as part of one config- 
uration operation. To complete the configuration 
action, the devices to be updated must continue to be 
reachable from the management station. Reconfigur- 
ing some devices may jeopardize the connectivity to 
other devices that need to be configured. Sequencing 
is the process of determining the order of updates to 
ensure a successful configuration. 


In this paper, we present one of the first studies 
of automated sequencing in the context of router? 
reconfigurations in IP networks. Many deployments 
today use an in-band access to the routers for configu- 
ration, where the control and data packets flow on the 
same network’. In this case, configuration of a router 
is done through a telnet to the router. Control (configu- 
ration information) and data go over the same physical 
and logical network, directed by the same routing 
tables. Parameters that an administrator can reconfig- 
ure include what we call critical and non-critical 
parameters. Changing critical parameters affects how 
the router computes and exchanges routing informa- 
tion with other routers, and consequently threatens 
continued connectivity to the router and other routers. 
Examples of critical parameters include routing proto- 
col parameters, like the Open Shortest Path First 


"Portions of this work were performed while all authors 
were at Bell Labs. 

2We use the term router to refer to any remotely config- 
urable network element. 

JA few routers in the network may have an out-of-band 
connectivity through the modem or console port of the 
router. Some establishments provide an out-of-band access 
to all routers in their network. 
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(OSPF) [10, 12] hello interval, router dead interval, 
authentication, and authentication key parameters, 
fundamental parameters like IP addresses and subnet 
masks, some access list configurations, etc. Non-criti- 
cal parameters, on the other hand, do not affect con- 
nectivity. Examples include parameters like the OSPF 
maximum paths and cost parameters. 


One can see that routers can be updated in any 
arbitrary order while changing non-critical parameters, 
or when an out-of-band access is available to all the 
routers. In this case, by out-of-band we mean access 
through the console or modem ports of the router. 
However, the order in which the updates are commit- 
ted (1.e., sequencing) becomes important: Administra- 
tors currently try to deduce a sequence manually from 
their knowledge of the network topology and their 
understanding of protocols. No automated techniques 
have been published for sequencing. 


In this paper, we study the problem of sequenc- 
ing when updating critical parameters, and present 
automated methods to compute an order in which to 
update the routers. We have implemented the tech- 
niques in Java, and conducted experiments that quan- 
tify the usefulness of our techniques and the conve- 
nience they can provide to the administrator. We 
demonstrate that our automated sequencing techniques 
speed up network configuration and make it more rell- 
able. 


The rest of the paper 1s organized as follows. The 
sequencing problem is explored in more detail in the 
next section. Our algorithms for sequencing router 
configurations are presented thereafter. The imple- 
mentation issues we encountered are presented in the 
implementation section which is followed by a section 
on experiments and results. We then conclude. 


Sequencing: The Problem and Assumptions 


To illustrate the problem of sequencing more 
concretely, consider the network in Figure 1. Let us 
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assume that all depicted routers are running the OSPF 
routing protocol, and are in the same OSPF Area. (See 
Appendix 2 for a brief description of OSPF.) Let us 
assume that we want to change the OSPF authentic a- 
tion parameter. The OSPF authentication parameter 
should be set consistently for all routers in an OSPF 
area to ensure that they exchange routing information. 
If during configuration, we update router R before all 
routers in the set S, there is a possibility that routers in 
set S may become unreachable from the network oper- 
ations center (NOC)‘. Sequencing ensures that the 
routers that remain to be updated will continue to be 
reachable. In our example, updating all routers in set S 
before router R is a viable strategy. 


In our study of sequencing during IP network 
configuration, we make certain assumptions. We 
assume that the changes are being made to the routers 
from a remote centralized network operations center 
and that all routers are initially reachable. Further, the 
changes to be made are correct; 1.e., if the desired 
changes are committed to the routers, the routing will 
stabilize soon enough, based on the routing protocol 
behavior, and to a state as desired by the administrator. 
We refer to this as the correctness assumption. The 
reachability of routers can be affected by several fac- 
tors, some of which are outside the scope of any soft- 
ware tool (e.g., power failure at a router). Such issues 
are not considered as part of this study of sequencing. 
We assume that any unreachability during the configu- 
ration operation is only caused due to the configura- 
tion changes being made by the administrator. 


4We use the terms network operations center and manage- 
ment station interchangeably in the paper. 


NOC 
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Situations may arise where no sequencing 1s pos- 
sible. Remote configuration of a set of routers could 
be theoretically impossible, under some conditions, 
mostly stemming from asymmetric routing. Consider, 
for example, the situation in Figure 2. 


Assume that the routes are set up (e.g., via static 
routes) so that all packets can only move clockwise 
around the circle C of routers. If a critical update is 
made to any router in the circle of routers, one of the 
paths (either to the router from the management sta- 
tion, or from the router to the management station) is 
broken, disabling the ability to reach and configure the 
other routers in the circle. Such a routing topology is 
unlikely in practice, since most people use routing 
protocols that will adapt to network change. However, 
this gives some insight into the theoretical limits of 
remote in-band configuration and sequencing in gen- 
eral. For simplicity, we assume in the following dis- 
cussion that the routing in the network is symmetric 
(although such a restriction 1s not really needed for all 
our techniques), and elaborate more on such situations 
later in a section on asymmetry in routing. 


An alternative approach to the problem of 
sequencing 1s to have built-in support in all routers in 
the network for scheduling configuration changes. 
This means that the software running on the routers 
(e.g., the Cisco IOS) must support the scheduling fea- 
ture. In this scenario, the required configuration 
changes are uploaded to the router and a timer 1s set 
which determines when the changes are executed. 
Ongoing work in the area of scheduling and scripting 
management information bases [8, 9] can aid in this 
problem. However, routers in the marketplace do not 


Figure 1: A network of routers and its network operations center (NOC). 





Figure 2: Circular dependencies and sequencing. 
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provide such a feature. Given that in-band remote con- 
figuration is becoming increasingly popular for IP net- 
works, developing smart automated sequencing tech- 
niques is essential. 


Context for our Study 


This study of sequencing arose in the context of 
an IP network configuration tool [5, 13] that was built 
at Bell Labs. The tool helps in configuring IP routers, 
treating them as a part of the network, rather than con- 
sidering them as stand-alone devices. The tool ensures 
consistency of configuration and updates several 
routers as part of a configuration operation, if neces- 
sary. The protocol used to contact the routers for con- 
figuration is orthogonal to the issue of sequencing. 
Although there are several proposed standards for 
management that can be used for configuration [3, 4], 
most deployed routers today provide a command-line 
interface for configuration. Configuration commands 
typically take effect as soon as they are executed. 


Our network configuration tool provides a GUI 
to the administrator for configuration, and hides all its 
interactions with the network and the routers from the 
administrator. To interact with the routers for configu- 
ration, the tool stores router passwords in an encrypted 
form in its database, and uses these passwords to login 
to the routers. 


Before we present our sequencing techniques, we 
introduce the notion of access to a router via indirect 
telnet. 


Access via Indirect Telnet 


An indirect telnet is essentially a proxy telnet. 
The basic idea in indirect telnet is to access a (possibly 
unreachable) router R through an intermediate router 
N, by first telnet’ing to the intermediate router NV and 
then initiating a telnet to router R from router N. 


The usefulness of indirect telnet is best illus- 
trated by the case where routers R and N have a 
directly connected interface. Since directly connected 
IP-configured interfaces are visible to each other, if 





(a) Forward traceroute tree 
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router NV is reachable, then we can access router R 
through its neighbor NV. Such a technique for reaching 
routers 1s sometimes used by administrators while 
configuring routers manually. The concept can be 
extended to hop through several intermediate nodes 
en-route to a final destination router. Our tool imple- 
ments an automated 1-hop indirect telnet scheme. 


Sequencing Techniques 


We present two main techniques for sequencing. 
The treegen method 1s a tree generation family of tech- 
niques. The treegen method deduces the routing topol- 
ogy to determine the order for updates. The reachable 
method is a sequential technique that uses mdirect tel- 
net (described in in the previous section) in an inter- 
esting way to perform sequencing. For the following 
discussion, we refer to the technique that chooses to 
update routers in no particular order (1.e., effectively, 
randomly) as the heuristic random; unlike our tech- 
niques, the random technique does not aim to update 
all the routers. 


The treegen Family of Techniques 


The basic purpose of the treegen family of tech- 
niques is to deduce the routing path tree to the nodes 
that need to be updated, and perform the configura- 
tions bottom-up in this tree. Such a tree can be built by 
performing traceroutes to the routers that need to be 
updated. The output of the traceroutes can be put 
together (using commonly known techniques, e.g., as 
done in [2, 7] for a different problem) to obtain a 
directed tree of the routers which depicts how packets 
are routed to these routers. 


A simple such traceroute tree is shown in Figure 
3a; it may have been constructed through a traceroute 
from the network operations center (NOC) to router 7s 
followed by traceroutes to routers 7, and r;. The tech- 
nique assumes that the part of the routing tree not yet 
updated will be stable for the time required to perform 
the configurations. Multiple routes to the same host 
may exist, but choosing one is generally sufficient. 





(b) Getting the reverse traceroute tree 


Figure 3: Forward and reverse traceroute trees. 
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Once the tree is obtained, the sequence is specified 
from the leaves to the root of the tree; i.e., all nodes 
reachable from a node must appear before the node in 
the sequencing order. Such an order can be obtained 
from a tree, e.g., by using depth first search techniques 
[1]. In Figure 3a, 73, 74, r2, 75, 6,71, and rs, 73, 16; Ya, 
ry, Yr; are valid sequences. We call our technique 
described above as the traceroute method. 


While doing a traceroute, if there is no response 
to a probe within the specified timeout, a ‘“‘*”’ output 
is observed. It is not uncommon to see ‘‘*”’ outputs for 
all probes, especially at higher time to live values (1.e., 
for probes reaching farther from the host). This slows 
down the traceroute method and also makes it less reli- 
able. Routers on the network with out-of-band connec- 
tivity could be exploited in the sequencing process. 
We now present some enhancements to our traceroute 
method that deal with some of these issues. 


The reverse-traceroute Method 


If we have access to the routers (1.e., the router 
passwords, which are needed for configuration), we 
can perform a traceroute from the routers to the man- 
agement station rather than to the routers from the 
management station. These traceroutes can be put 
together to deduce the tree from the NOC to the 
routers. If the routing is symmetric, such a method that 
does traceroutes from the router would give the same 
tree as the generic traceroute method. In this case, 
however, we have the advantage that we can do a 
restricted traceroute, possibly looking only at the next 
hop on the routing path from the router to the manage- 
ment station. These {router, next hop} pairs can be put 
together to get the full tree. For the example in 3a, the 
method to obtain the reverse traceroute tree by con- 
ducting one-hop traceroutes from each of the routers is 
depicted in Figure3b. 


In theory, we need to examine the traceroute 
from each router until we hit a router that also needs to 
be updated. This ensures that we have the complete 
dependency between the routers that need to be 
updated. In most cases, however, the set of routers to 
be updated is contiguous and looking at the next hop 
suffices. If access to all routers exists, the routing table 
entries can be examined in lieu of a traceroute. A 
traceroute is a simple way to get the route information. 


The forward-reverse Method 


The safer method for constructing the tree is to 
perform both the traceroute and reverse-traceroute meth- 
ods. This will avoid depending on the symmetric rout- 
ing assumption. If we get a tree (or a directed acyclic 
graph [1]) by putting the trees derived from the tracer- 
oute and the reverse-traceroute methods together, the 
sequencing order can be determined. A cycle indicates 
a possible generic problem in determining a sequenc- 
ing order. If such a situation is due to multiple routing 
paths in the network, there is no problem, since the 
routing path will readjust, if needed. If the situation is 
due to forced asymmetric routing (as described in 
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Figure 2), there is a problem. However, this is a patho- 
logical example and most installations do not design 
their networks this way. 


Other Generalizations 


There can be several possible start nodes for ini- 
tiating the traceroutes. It is necessary to have access 
(1.e., login) to each of the routers and machines from 
which a traceroute is to be performed. Furthermore, 
there must be a path to the start nodes that does not go 
through other nodes that are to be updated. The fol- 
lowing nodes are examples of candidates for start 
nodes: the machine in the network operations center, 
OSPF area border routers, and routers with out-of- 
band (e.g., modem or console) connections. 


Let S be the set of nodes to which changes need 
to be made. Let SN be the non-empty set of possible 
start nodes. The output of the sequencing method is an 
ordered list of elements: (5;,s7;), where s; is a node 
from set S and sn; is a node from set S'V. The implica- 
tion is that configuration changes must be made in the 
order specified by the list, and a telnet to node s; must 
be accomplished from node sn, (1.e., via an indirect 
telnet strategy). 


Conceptually, we build traceroute trees from 
each of the start nodes. Each tree can either encom- 
pass all nodes in the network, or only the nodes not 
covered by already created trees. The dependencies 
can be computed from these sets of trees using stan- 
dard graph theoretic approaches as in [1]; we omit 
these details from this paper. This generalization lends 
to interesting extensions of the basic approach, and 
techniques that mix pure in-band and pure out-of-band 
configuration approaches. 

Optimizations 

The treegen family of techniques lend themselves 
to various optimizations. Nodes that do not depend on 
each other can be updated in parallel to improve per- 
formance. The traceroutes in the reverse-traceroute 
method can be done in parallel to improve perfor- 
mance. In some cases, instead of deducing the routing 
topology by examining the routing tables or perform- 
ing traceroutes, an analysis of the physical topology 
can indicate the routing topology (e.g., a simple line 
network). 


The reachable Technique 


Unlike the treegen family of techniques, the 
reachable technique merges the processes of determin- 
ing the update order and the router configuration 
updates. At the ith iteration, the reachable technique 
determines the next router to update, updates this 
router, and then determines the i+ 1st router to 
update. The choice of the next router depends on the 
current state of the network. The reachable technique 
assumes that a one-hop indirect telnet access is avail- 
able to reach a router. 


The reachable algorithm updates reachable 
routers via telnet. If there are no remaining reachable 
routers, the process switches to indirect telnet via a 
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router whose neighbor is reachable. (Indirect telnet 
was explained in an earlier section.) More formally, let 
S denote the set of all routers to be reconfigured, let U 
denote the set of already reconfigured routers, let TBU 
denote the set of routers that remain to be reconfig- 
ured, and let R denote the set of routers that are cur- 
rently directly reachable from the management station. 
Initially, VU = ®, and TBU= R=S. The set R can be 
determined at any point in time via pings from the 
management station, and 7BU is always equal to 
S — U. Algorithm reachable uses the following logic to 
determine the next router to reconfigure. If there is a 
router to be reconfigured that is directly reachable 
(i.e., in set R), it updates that router. If there is no 
directly reachable router (i.e., R () TBU = ®), reach- 
able reconfigures via indirect telnet a router from set 
TBU whose neighbor is reachable (1.e., the neighbor is 
in set R). Otherwise, it declares failure. After configur- 
ing a router, it updates sets U and TBU, recomputes set 
R, and continues, as long as TBU # ®. 


An interesting property of the reachable algo- 
rithm is that under most conditions, 1t will not fail. In 
other words, it will always find a router to update. We 
formalize this notion in Appendix 1. 


Implementation 


We implemented the techniques described earlier 
to obtain measurements on their efficacy. The imple- 
mentation was done in Java. For our prototype ver- 
sion, we adapted a Java telnet implementation [6] 
based on Java sockets to contact the routers. The 
router passwords were available to log into the router 
and perform configuration. We wrote our sequencing 
routines as part of the IPNC tool [5, 13]. Our imple- 
mentation allowed us to determine information about 
neighbors to the routers, and the ability to do one-hop 
indirect telnet, perform traceroute from the routers, 
generate the correct router commands and perform the 
required interactions with the routers. 


The treegen Technique 


We implemented the reverse-traceroute method 
from the treegen family of techniques. We imple- 
mented two variants of the method. In the first one, 
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which we call trace-parallel, after the order is deter- 
mined, the routers are updated with maximum possi- 
ble parallelism; in other words, at any stage, all leaves 
of the tree are updated in parallel. In the second vari- 
ant that we call trace-seq, the routers are updated 
sequentially. In both cases, the traceroutes from the 
routers are executed in parallel to obtain the order of 
update speedily. We restricted ourselves to three hops 
of output from traceroute. The routers we used did not 
allow an easy (non-interactive) way of specifying the 
maximum number of hops, and this had to be worked 
in from our implementation. 


The reachable Technique 


Implementation of the reachable technique 
required indirect telnet support. We performed pings in 
sequence, and hope to move to a parallelized imple- 
mentation soon. Coming up with a suitable timeout to 
account for routing stabilization is non-trivial; we 
approximated with retrying the configuration opera- 
tion after waiting five seconds upon a failure, and this 
worked for us. We elaborate more on this later in a 
section on Running Time Measurements. 


Experiments and Results 


Experiments were conducted on networks of var- 
ious topologies. Measurements on the three basic 
topologies (line, tree, and circle) shown in Figure 4 are 
reported below. The other topologies we experimented 
with were made up of these basic topologies, as are 
most networks. The main results from other topologies 
we tested were in agreement with what we present 
here. 


The routers in our experiments were Cisco 
routers (25xx, 26xx, 36xx, and 40xx series), running 
various versions of IOS (all version 11.3 or higher). 
The interfaces were a mix of ethernet, high speed 
serial, serial, and token ring interfaces. For our experi- 
ments, we enabled the routers to run OSPF, put all 
routers in the same OSPF area, and changed the OSPF 
area authentication parameter on the routers. (See 
Appendix 2 for a brief description of OSPF.) In this 
case, a change at a router does not immediately make 
itself or other routers unreachable. The protocol! takes 
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Figure 4: Network topologies for reported experiments. 
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some time to settle depending on other parameters 
(like hello and dead intervals) that we could change. 
Keeping the hello interval fixed at a low value (e.g., 
1-2 seconds), a high value of the dead interval holds 
the network routing virtually unchanged for a long 
time, while a low value for the dead interval propa- 
gates the effect of the parameter changes quickly. For 
each of the methods random, trace-seq, trace-parallel, 
and reachable, we obtained measurements on (1) the 
number of routers that were successfully configured, 
(2) the time it took to determine the sequence order 
and (3) the time it took to perform the configuration. 
In the case of reachable since the sequencing is inter- 
woven with the configuration, we only obtained one 
number: the total time to complete the reconfiguration. 
We now present our results under various categories. 


Number of Routers Updated 
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routers. This was true for all values of hello/dead 
interval we tested. 


An interesting situation in this case is with the 
random technique. With random, since the routers are 
chosen in an arbitrary order and no sequencing is per- 
formed, some of the routers may fail to get configured. 


Depending on the hello and dead intervals and 
the random order, up to 60% of the routers failed to 
get updated by random. An illustration of this effect is 
shown in Figure 5, which presents a plot for a random 
sequence. The plot in the figure must be taken with 
some caution. By using automated techniques rather 
than manual interaction with the router for configura- 
tion, quick router configurations were possible. (More 
details of the time for configuration appear below.) 
The effect from Figure 5 may be vastly different for 
manual configurations in that many more routers may 


The trace-seq, trade-parallel, and reachable tech- 
niques were always successful in updating all the 


feo [tet | ae Le ta | tee [to [te 
[ece-seq | 16.70 | 18.92 | 3562 | 27.86 | 19.63 | 4749 | 2352 | 15.06 | 3858 
wooeparail | Tare | W7ei_| ser | 7rar | ei | aor | Teas | aa | 7005, 
[reachable | 


Table 1: yer time to perform sequencing and configuration for the three topologies and the three techniques; 


tseq 1S the time to determine the sequence, f,,,, 1s the time to do configuration, and ¢,,4; 1S the total time taken. 


The times are for all routers in the network to be updated. All units are in seconds. 
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Tree (10 routers) | Circle (8 routers) | 
rraoeseq | 231 | 052 | 1.79 | 343 
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Table 2: Standard deviations in time to perform sequencing and configuration for the three topologies and the three 
techniques; 0g 1s the standard deviation in the time to determine the sequence, 0,,,¢ is the standard deviation in 


the time to do configuration, and 0,,,,; 1s the standard deviation in the total time taken. The times are for all 
routers in the network to be updated. 
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fail to get updated, since each router takes longer to be 
updated when done manually. Second, the random 
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Table 3: aveniee time (per router) to perform sequencing and configuration for the three oRalenies and the three 
techniques; ft... 1s the time to determine the sequence, f,,,7 1s the time to do configuration, and ¢,.;,; 1s the total 
time taken. These numbers are derived from Table 1 by dividing the total time by the number of routers, and is 
presented here for convenience. All units are in seconds. 
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order is significant, and the plot implicitly says more 
about the time for the routing to settle rather than 
about the random technique itself. Third, the graph 
may look different if a parameter other than dead 
interval were chosen for the x axis. We chose the dead 
interval since it is a good control variable for how 
soon the configuration changes affect the routing. 


10 


Min. #routers successfully updated 


Dead interval 
Figure 5: Minimum number of routers (out of 10) 
updated by random as a function of OSPF dead 
interval using a hello-interval of 1 second on the 
tree network. 


Running Time Measurements 


Tables 1-3 show various statistics related to the 
time taken by our techniques for determining the 
sequence and performing the configuration changes. 
All values are derived from 3-4 runs. (We did many 
more runs, but they used different topologies and dif- 
ferent number of routers. We observed similar results 
from those experiments as reported here.) Table 1 
shows the total time for all routers in the network to be 
updated and Table 2 shows the standard deviation of 
the observed times. For convenience, we divide the 
total time numbers from Table 1 by the number of 
routers in the network to get per router timing num- 
bers in Table 3. The topologies have different number 
of routers (eight each for the line and circle, and ten 
for the tree); in addition, the routers and interfaces 
used in the topologies were also slightly different. 


The first interesting result is that automation 
makes the total time taken to do the configuration very 
small (on the average, 0.6-2.23 seconds per router for 
trace-parallel as seen from Table 3). This enables rapid 
reconfiguration and makes our tool very useful. As a 
point of reference, our (informal) study shows that 
manual reconfiguration of authentication by experi- 
enced administrators may take about 30-45 seconds 
per router (excluding times for sequencing), resulting 
in 5-7.5 minutes for reconfiguring 10 routers. This 
estimate for manual configuration excludes the time to 
generate the configuration commands, but includes the 
time to type in the commands at the router’s 
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configuration prompt. In the case of OSPF authentica- 
tion, this typically translates to 4-6 commands per 
router. In the automated case, the commands are auto- 
matically generated and sent to the router by our tool, 
and that time is included in the reported numbers. 


The automation also lends to easy parallelization 
of the configuration operation, when possible. The 
performance improvement resulting from the paral- 
lelization is visible from the ¢,4,, value for the trace- 
parallel technique, which shows a 34.6% and 68.8% 
improvement over the corresponding ¢,,,¢ time for the 
trace-seq technique for the circle and tree topologies, 
respectively. No parallelism is possible in the case of 
the line. The traceroutes in the case of the trace-seq 
and trace-parallel methods were done in parallel, and 
the t,,g time is dominated by the amount of time it 
takes to get a 3-hop output from the traceroute done at 
the router (since all other processing on this output is 
minimal and in-memory). For random, when it is suc- 
cessful in completing the configuration, the t,o, = total 
and is approximately the ¢..,¢ value for trace-seq. 
From the table, it is difficult to break up the time taken 
for configuration alone by reachable. The time for non- 
configuration activities (including pings, timeouts, and 
indirect telnet) for reachable can be estimated, how- 
ever, by 

(trotai(reachable) — t,,,/(trace-seq)/t,,,.;(reachable) 
to be 67-73% for the three topologies. 


The comparatively low values for standard devi- 
ation from Table 2 suggest that the timing measure- 
ments did not vary significantly. 


Interestingly, the timing values for trace-seq and 
trace-parallel were rather predictable and independent 
of the hello/dead interval values (and hence, the time 
taken for route changes). However, the same was not 
the case for reachable. The time for reachable depends 
on the order in which the routers are updated, and how 
soon the routes change. In our implementation of 
reachable, the order in which the routers were tried for 
update depended on the hash function used by Java 
(since the router names were initially put into a hash 
table). Figure 6 shows how the time for configuration 
for ignored: reachable changed with the dead interval in 
the network. The increase in time when the routes 
change more rapidly (i.e., at lower values of dead 
interval) comes from the increase in the number of 
indirect telnets performed and the timeouts when per- 
forming pings (when the routing is settling down). As 
mentioned in Appendix 1 and the implementation sec- 
tion, in our experiments, when there was a failure, 
retrying after a timeout of five seconds provided a 
router as per the condition in Proposition | in 
Appendix 1. 


In rare circumstances (in two cases out of several 
tens of tests we ran), we observed that a ping to a 
router succeeded, but the routing changed between the 
ping’s success and completion of the configuration. 
Such occurrences may be more frequent in some net- 
works. However, it does not matter to the reachable 
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technique from the point of view of correctness, since 
it assumes that the router was unreachable. It does, 
however, increase the running time of the technique. 
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Figure 6: Time taken by reachable to do configuration 
as a function of the dead interval using a hello 
interval of 2 seconds on the tree network. 


Asymmetry in Routing 


In most of our topologies, the routing was sym- 
metric. In the circle case, both paths around the circle 
could be taken by the packets. We did not explicitly 
manipulate the routing tables, and left the cost of both 
paths to be the same (for OSPF purposes). The 
dynamic nature of the routing combined with the 
sequencing techniques we used resulted in no configu- 
ration problems for the topology using our techniques. 
In our experimentation, we concentrated on topologies 
and scenarios that we felt were representative of real 
networks, and hence did not choose configurations 
like in Figure 2. 


Conclusions 


In this paper we have studied the problem of IP 
network configuration. In particular, for the first time, 
we have studied the issue of sequencing updates of 
critical parameters in a network of IP routers. We 
have presented several techniques to sequence the 
configuration of the routers to ensure continued con- 
nectivity to the routers during the time that it takes to 
perform the configuration. We have implemented our 
techniques, and have presented experimental results 
on the validity of the techniques and measurements of 
their performance. We have shown that automated 
configuration significantly speeds up time for configu- 
ration (from an estimated 45 seconds per router while 
done manually to 0.6-2.3 seconds when automated), 
and our sequencing techniques make configuration 
fast and reliable. Our automated techniques also make 
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the configuration process convenient for the adminis- 
trator. Some of our sequencing techniques are also part 
of the IPNC tool [5]. We believe that this significantly 
advances the state of the art in a field dominated by 
manual configuration of routers. We believe that such 
tools and techniques for automated network configura- 
tion will prove valuable to the users. 


The network configuration tool with appropriate 
enhancements is currently available from Lucent 
Technologies (http://www.lucent.com) and ISPsoft, 
Inc. (http://www. ispsoft.com). 
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Appendix 1: Success Criterion for the reachable 
Algorithm 


The success criterion for the reachable algorithm 
is based on the following communication property. In 
most cases, nodes that have already been reconfigured, 
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and that are (topologically/logically) connected will 
start communicating with each other, and be reachable 
from each other. Given the correctness assumption 
from earlier, such nodes will continue communicating 
with nodes that do not need to be reconfigured and 
that are connected to them. Further, if any one of these 
nodes 1s reachable from the management station, all of 
them will be reachable. 


While it 1s not essential that the communication 
property hold true, it often is, especially for routing 
protocol parameter updates. In such cases, the follow- 
ing proposition holds. 

Proposition 1: Let the communication prop- 
erty be true. If there remain a set of routers to 
be reconfigured, there always exists a router in 
this set such that either that router or one of its 
neighbors is reachable. 


Proposition 1 implies that when the communica- 
tion property is true, algorithm reachable will always 
be successful in its reconfiguration operation. The 
validity of Proposition 1 is not hard to deduce; we pre- 
sent an informal proof below using Figure 7. 


Let a be a type of node that does not need to be 
reconfigured (i.e., a node of type a é S), b, be a type 
of node that has already been reconfigured (i.e., a 
node of type 5, € U), and bj», be a type of node that 
has not yet been reconfigured (1.e., a node of type 
by, € TBU). (We use the set definitions from the ear- 
lier description of algorithm reachable for simplicity in 
exposition.) Take any node x from set 7JBU (1.e., of 
type 5,,,) and conceptually trace the routing path? 
from the NOC to x. (We do not need to know this 
path; it is only used for the proof.) Map the nodes on 
the path to their types; for the example in Figure 7, we 
get a string of the form ‘“ab,b,abjy,q5,5;5,". By the 
communication assumption, the nodes corresponding 
to the initial string matching the regular expression 
““(a*b,,*)*” (in this case ab,b,a) are reachable from 
the NOC, and hence a neighbor of the first by, (i.e., 
the neighbor z of node y) is reachable. If the initial 
string ‘“‘ab,b,a’’ were empty, then the first node of 
type b,,, 1s directly reachable. Note that it is not the 
node x we selected initially in our proof that is 


The term routing path is used loosely here. A formal proof 
would consider the path packets would take in trying to 
reach 6,,,,, even if packets do not complete the journey. 


5 See y 


a ee ee 
a b 


ul tbu 


Figure 7: Reachability snapshot for Proposition 1. 
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reachable; all the proposition says is that there exists 
some node that 1s reachable or whose neighbor 1s. 


The communication property does hold for the 
update of most routing protocol parameters. Even in 
such cases, the routing takes some time to settle down, 
and the communication property holds after the rout- 
ing has settled down. The algorithm has to incorporate 
timeouts and retries to take care of this phenomenon. 
Typically, the time taken for the routing to settle down 
for a given network after each reconfiguration can be 
estimated to be some time ¢. If there are routers 
remaining to be updated, and we are unable to find 
any, retrying once after waiting for time ¢ should pro- 
vide a router as per Proposition 1. Although ¢ can be 
pretty large in theory, in practice it is reasonably 
small, and the number of times you need to wait is 
expected to be infrequent. Another possible problem 
that could arise in practice is that a router is reachable 
but becomes unreachable before the configuration 
action is completed. We describe our experiences with 
these issues in time measurement section. 


Appendix 2: Routing and OSPF - A Brief 
Overview 


Routers are critical network elements of IP net- 
works. The most important function of an IP router is 
to route data packets. A forwarding table inside a 
router is used to decide which route a packet should 
take. There are several ways to add entries to a for- 
warding table. Typically, a protocol builds this table 
dynamically. These protocols are called routing proto- 
cols. A routing domain is a collection of routers and 
all routers in a routing domain run the same routing 
protocol. One or more such routing domains constitute 
an Autonomous System (AS). A unique number iden- 
tifies this AS to the rest of the autonomous systems. 
This number is called AS number. Routing protocols 
designed to run among AS are called Exterior Gate- 
way Protocols (EGP). Routing protocols designed to 
run inside an AS are called Interior Gateway Protocols 
(IGP). One of the most popular IGPs is the Open 
Shortest Path First (OSPF) routing protocol [10, 11]. 


OSPF is a hierarchical routing protocol. It can 
run in the entire AS. The AS can be divided into 
OSPF areas. A unique number identifies each area. An 
area identified by number zero is also called the OSPF 
backbone. Grouping of routers in an area is a network 
planning and design issue, which is beyond the scope 
of this paper. A router is an Area Border Router 
(ABR) if it is part of more than one area. Entries in a 
forwarding table, 1.e. routes, are generally aggregated 
at the area level by the ABR so that the processing 
burden and storage requirements of the routers in the 
other areas are reduced. For OSPF to work correctly, 
at least one ABR in an area must be connected to the 
OSPF backbone. 


Some OSPF parameters used in this paper are 
described below. 
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Hello Interval. A router running OSPF sends a 
“hello” packet periodically to inform its neigh- 
bors that it is alive. The fixed interval between 
consecutive hello packets is called the hello 
interval. 

Router Dead Interval. If a router does not 
receive a hello packet during a fixed time inter- 
val from a neighbor, it declares that neighbor 
dead. This interval is referred as the router 
dead interval. 

Area Authentication. A technique used to 
ensure that the router sending OSPF informa- 
tion is trusted and the OSPF packet has not 
been altered. 
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ABSTRACT 


ND 1s a software tool developed by the Computing and Information Services Network Group 
at Texas A&M University (TAMU) to aid in the engineering and operation of the campus network. 
This tool was developed in response to the tremendous growth of the TAMU campus network over 
the last ten years. ND is designed to provide high-level application functionality while retaining 
the power and flexibility of a low level tool. It integrates remote network device analysis (via 
SNMP) with network defining databases (via SQL). ND is written in Python [1] and contains 
custom modules for interaction with SNMP and MySQL [2]. 


Introduction 


ND 1s a tool that was developed to aid in the 
Operation and, more importantly, the engineering of 
the TAMU campus network. With a network of more 
than 25,000 nodes and over 1000 network devices, a 
tool was required that could scale to a high level and 
provide useful functionality without the need to 
repeatedly develop special purpose scripting based 
solutions. 


One of the major design goals of ND was to have 
a command line interface (CLI) syntax that was acces- 
sible remotely via Telnet/SSH. This is a very impor- 
tant feature because this type of remote access is the 
lowest common denominator that can usually be found 
on almost any host or network device in the field. It 
requires neither a graphical interface nor the installa- 
tion of special software. In addition, the CLI syntax 
provides a common command structure independent 
of different vendors’ software implementations due to 
the use of standard SNMP MIBs. 


Another design goal was to integrate ND into the 
Unix environment in order to leverage and adopt some 
of the features of Unix that have given it the ability to 
scale. This integration also made it simple to use a 
number of freely available support tools and technolo- 
gies such as Python, SNMP and SQL. The resulting 
synergies have resulted in a very powerful and useful 
tool. 


This paper presents a background section on net- 
work management applications that explains why ND 
needed to be developed, a design section covering the 
architecture of ND, a section on how ND 1s used, and 
finally a section discussing what the resulting benefits 
have been. 


Background 


A quick search of the World Wide Web reveals 
that there are a remarkable number of packages avail- 
able both commercially and as open source that pur- 
port to perform “network management’’. This leads 
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one to ask, ‘““Why build yet another network manage- 
ment tool?” 


Network management applications have evolved 
into four major categories: event management, perfor- 
mance management, policy management and finally 
‘““generic”’ network management. Event management 
applications are based on the monitoring and manag- 
ing of network events such as the reception of SNMP 
traps and up/down state information based on ICMP 
echo request/replies (ping). Two tools in this category 
are Big Brother [3] and Micromuse’s Netcool/ 
Omnibus [4] product. Performance management tools 
collect, monitor, and present network performance 
information (usually collected via SNMP). InfoVista 
[5] and MRTG [6] are examples of tools in this cate- 
gory. 

Policy management mainly refers to proprietary 
applications developed by network equipment vendors 
that attempt to manage complex network equipment 
functionality (such as quality of service and security) 
via a policy abstraction. Cisco and 3Com are examples 
of vendors that have developed tools of these types. 
Finally, the category of “generic” network manage- 
ment applications is the most recognizable one. Appli- 
cations of this type tend to center around a Graphical 
User Interface (GUI), which displays information 
about network topology, network elements and status. 
Commercial versions of these tools typically incorpo- 
rate an element manager, which provides a GUI that 
allows in-depth control of a network element. HP’s 
OpenView [7] and Scotty/Tkined [8] are examples of 
this type of application. 


One of the weaknesses of network management 
applications as a whole is that they tend to address 
only the operational aspect of “network manage- 
ment’’. Those who engineer networks tend to eschew 
these applications for their work because of their con- 
strictive nature; while GUIs are useful for displaying 
complex information, they are typically implemented 
in such a way that limits the ability of users to perform 
complex operations on medium to large sets of 
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network elements. They also do not take into account 
nor provide support for the specific network manage- 
ment practices that may be in use at a particular site 
(naming conventions, addressing conventions, net- 
work layouts, port numbering pattems, organizational 
structure, etc.). 


True to their nature, network engineers tend to 
focus on network management tools rather than net- 
work management applications as categorized above. 
These tools are typically programming/scripting lan- 
guages (Perl, Python, Expect, etc.), SNMP MIB 
browsers, libraries and standalone tools (SNMP++, 
UCD Snmp, etc), and databases (MySQL) The result 
is that engineers build special purpose applications on 
a case-by-case basis that bridge this gap between the 
tools and the applications. This software tends to be 
powerful and effective but special purpose and diffi- 
cult to maintain and extend. 


ND was developed to specifically address this 
problem; it has been designed to be a powerful tool 
built on SNMP MIB functionality and SQL databases 
that provides a useful and friendly application inter- 
face. While the Simple Network Management Execu- 
tive [9] (SNMX) package provides a powerful script- 
ing and SNMP tool structure and is thus closest in 
nature to ND, ND is unique in the level of functional- 
ity it provides in the form of a flexible tool. 


ND Architecture 


Python 


ND is written in Python and contains custom 
modules for interaction with MySQL and SNMP. 
Python was chosen for its ease of programming and its 
modular and object-oriented design. There are over 40 
modules in ND (approximately 10,000 lines of code) 
grouped into 8 families. Table 1 summarizes the mod- 
ule families. The module families closely follow indi- 
vidual MIBs (both standard and proprietary). Several 
of the standard Python modules have been customized 
while another has been newly written. Table 2 summa- 
rizes the modified or created Python modules. 


To assist in creating new modules, a Python class 
was written from which all modules are sub-classed. 
The parent class provides methods for parsing user 
input, redirecting output to pipes or files, providing a 
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command-line history, and providing a mechanism for 
methods common to all modules, e.g., help. 


ND contains an extensive help system that is 
available at two levels. First, at each ND prompt, a 
question mark can be entered which will list all the 
commands available in that module. Second, the help 
command can be issued with a specific command 
which will cause the syntax and description of 
thecommand to be displayed. 


Module Description 


mysql Modified the initial connection 
code to use MySQL conf files 


readline | Modified to allow for user defined 


history files 

A new module that allows Python 
access to the SNMP++ and UCD 
SNMP libraries 


Table 2: Modified/new python modules. 


snmp++ 





Databases 


ND makes extensive use of SQL tables and uses 
the MySQL server as a backend. Table 3 lists the 
major categories of SQL tables. Originally, ND used 
an early version of msq] and db, which essentially had 
the ability to store simple flat files. To ensure that the 
database made sense from a higher network design 
perspective, ND was designed to keep databases con- 
sistent and free from typical types of errors as might 
be prone with manual input. This includes sanity 
checking of record relationships on the creation of 
new records and automatic generation of record id 
numbers (functionality that is now found in MySQL). 


The Fiber Circuits and Twisted Pair Drops cate- 
gories are the result of extensions that have been made 
to ND over time. These are accessed via the equiva- 
lent of small applications within ND that specialize in 
the management of the campus network’s fiber optic 
plant and twisted pair wiring plant. This functionality 
was added to ND due to the CLI infrastructure ND 
provided and due to the usefulness of sharing this 
information with the rest of ND. Fundamentally, these 
small applications provide support for the processes 
used at TAMU related to plant management and 
demonstrate the ability of ND to support site-specific 
business practices. 





Module Family Description MIB 
fms 3Com FMS Repeater MIBs Proprietary 
ost Alcatel Switching MIBs Proprietary 
bridge Bridge MIB RFC 1286 [10] 
fr Frame relay MIB RFE.1315 (11) 
repeater Repeater MIB RFC 1516 [12] 
rmon RMON MIB RFC 1271 [13] 
snmp Basic SNMP Query RFC 1213 [14] 
netdb Database applications n/a 


Table 1: ND module families. 
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Table Category Description 


Various support data such as 
campus buildings and depart- 
ment descriptions. 


Support 


Defines fiber circuits 

Defines each installed drop on 
campus. 

Contains basic information 
about backbone devices 


- Fiber Circuits 
Twisted Pair 





Table 3: Database architecture. 


SNMP 


SNMP is a fundamental aspect of ND [15]. Since 
Python contains no built-in support for SNMP, a new 
module was created. Initially, a set of functions that 
used the system method of the OS module was used to 
interface (externally as child processes) with the UCD 
SNMP [16] set of utilities. This proved inefficient and 
more importantly, resulted in a security risk since 
community strings would appear in the process table 
as arguments. 


A search for lower-level tools was initiated and 
the SNMP++ [17] toolkit was discovered. This toolkit 
provides a very efficient interface into the SNMP 
GET, PUT and FABLE functions but it does not have 
any provision for parsing MIBs. For this reason,. the 
interface to the UCD snmptranslate function was kept 
but an OID translation cache was added as one of the 
SQL tables. This cache all but eliminated the ineffi- 
ciencies of using an external program. 


It should be noted that the SNMP v1 protocol is 
insecure [18] by its nature since authentication tokens 
(community strings) are sent in plain text. Steps 
should be taken to guard against others snooping traf- 
fic on the network and discovering the community 
strings, which may provide the ability to change the 
configuration of a device. At TAMU, the campus net- 
work has been engineered in order to prevent eaves- 
dropping. 

Performance Issues 


Since Python is used primarily to implement the 
user interface, the performance of ND is more closely 
tied to the performance of the underlying SNMP and 
MySQL toolkits rather than Python. It has turned out 
that the load on the MySQL database server, even with 
a network as large as TAMU’s, has not been shown to 
be a problem. 


The real bottleneck has turned out to be the net- 
work devices response times to SNMP _ queries. 
Depending on the CPU and memory of the network 
device, it may take tens of milliseconds to respond to 
an SNMP query. When large tables need to be tra- 
versed (which must be done serially under SNMP v1), 
this can result in a large delay before an ND command 
completes. 


Another performance related issue is how ND 
reacts to poor network conditions. SNMP is a stateless 
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UDP protocol; reliable communications to a network 
device must be ensured by the SNMP software. Typi- 
cally, when SNMP responses fail to retum to an 
SNMP management agent, the agent will retransmit 
the SNMP request. Under poor network conditions, it 
is important to be able to control this behavior based 
on the desires of the user and the severity of the net- 
work conditions. ND provides timeout and retries 
parameters that control how long it takes for an indi- 
vidual SNMP request to timeout and how many time- 
outs are allowed before communication with the 
device 1s considered to have failed. 


Command Structure 


ND is a hierarchical, command-line interface 
program. We made this choice so that shell scripts 
could be written to automate complex and repetitive 
tasks. Also, this tool is frequently used in the field 
where the only available interface is a Telnet/SSH ses- 
sion. 


The hierarchical nature of ND follows a logical 
progression that we have found assists in problem 
tracking and security incident investigations. For 
example, the FMS family contains six sub-modules: 
address, ports, security, traps, users, and misc. Each of 
these in turn contains between five and ten individual 
commands. Similar structures exist for Alcatel, 
RMON, and bridge MIBs. 


The current position in the hierarchy is reflected 
in the ND prompt. When first invoked, ND starts at 
the root level. To navigate deeper into the hierarchy, 
the user needs only type the name of a module avail- 
able at that level. The quit command is used to leave a 
module and to return to a higher point in the hierarchy. 
The *..’ notation is used as a mechanism to reference 
another module’s commands without leaving the cur- 
rent module. When a hostname is required in a com- 
mand, the ‘!’ notation can be utilized to reference the 
host used in the previous command. This is very help- 
ful when the command is related to the same host but 
is different to the point that command line history edit- 
ing is not very helpful. 


The GNU readline toolkit, which provides com- 
mand line editing and history, has been incorporated 
into ND via the readline module found in Python. It 
has been modified to generate user defined command 
line history files, which allow the history to be persis- 
tent between ND sessions. 


There are a number of commands common to 
most ND modules: list, show, set, add, delete, enable, 
disable and help. These commands have basically the 
same syntax but are tailored to the functionality of the 
particular module. The commonality of the syntax 
provides a user with a good base understanding of 
how commands within any module work. They need 
only use the help command to find the specific syntax. 


One of the most powerful features of the com- 
mand line interface 1s the ability to pipe output of any 
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command to a Unix shell command or to a file. This 1s 
accomplished by using the normal shell characters of 
‘|’, “>’, and ‘>>’ at the end of an ND command. As an 
indication of this level of integration, ND does not 
have any output paging mechanism. The user 1s 
responsible for piping the output of any command to 
the paging program of their choice (e.g., more). 


Using ND 


All TAMU network operators and engineers use 
ND. The types of tasks that each group performs are 
quite different but ND is flexible enough for both. 
These tasks include troubleshooting, administration, 
monitoring, and maintenance. 


Starting ND is as simple as typing nd. Before 
presenting the top level prompt of nd>, ND will exe- 
cute any commands found in */.ndrc. Typically, some 
top level set commands are executed in order to estab- 
lish certain operating parameters that ND needs. The 
set command allows you to specify the following: 


sqluser user name to connect to MySQL server 

sqlpass password to connect to MySQL server 

sqlhost host where MySQL server is running 

timeout number of seconds for SNMP timeouts 

retries number of retries for SNMP queries 

snmplimit | maximum number of rows to return in 
SNMP queries 

read SNMP read community string 

write SNMP write community string 

rows maximum number of rows to output 
before reprinting column headings 

debug sets the internal debug level for 
MySQL and SNMP 

historyfile sets the file to use for the history file 
between ND sessions 

echo toggles ND echoing user commands 


Another command commonly found in .ndrc 
files is the attach command. The attach command 
allows simple tags to be defined for long or complex 
host names and also allows for community strings to 
differ from the default. For example, hostnames of 
networking equipment at TAMU follow a set pattem 
but are often tedious to repeatedly type. 


Once in ND, the user need only type the name of 
a module to navigate deeper into the hierarchy and 
quit to navigate back up the hierarchy. The prompt 
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will change to indicate which module the user is cur- 
rently in. Within each module the help command pro- 
vides module and command specific help. There 1s no 
documentation or training material for ND other than 
the internal help system. 


As mentioned previously, the ND command line 
interface is integrated with the Unix shell. Most com- 
monly, users will need to take the output of commands 
and pipe them to an output pager of their choice. 
More experienced users will save the output to a file 
Or pass it into a grep command looking for a particular 
pattem. Expert users will pass the output to awk or 
possibly even an external Perl or shell script. In sup- 
port of this functionality, the rows parameter can be set 
such that no column headings are printed with a com- 
mand’s output. 


Rather than give an exhaustive explanation of all 
ND modules and commands (which space will not 
allow), it is more useful to show by example, how ND 
is used at TAMU day-to-day. The following examples 
show how ND 1s used in a variety of both operational 
and engineering related situations. 


The general format of common ND commands 
1S: 
<eommand? “host? <unit> <port> <options? 


The units and ports can be specified using a scalar, 
list, or range notation. 


Scenario 1: Ports 1, 3, 4, 5 and 8 of unit 1 ofa 
new 3COM FMS have recently been activated. Before 
the users are informed that the ports are available, the 
configuration of the ports must be verified. The com- 
mand shown in Figure 1 would be used. 


Scenario 2: Ports have become scarce inside one 
building. It 1s suspected that some ports that are allo- 
cated are not actually being used. Viewing only the 
ports on unit 1 that have a total frame count greater 
than zero would indicate which ports are actually 
being used. The ND command and corresponding 
output are shown in Figure 5. 


Scenario 3: Several users have called and com- 
plained that the network in their building has become 
unresponsive. It is suspected that someone is using too 
much bandwidth. By inspecting the frame counts for 
all ports on a device and then sorting the output, the 
ports using the most bandwidth are easily found. The 
output for this example is in Figure 6. 


nd-fms-ports> show fms-device-1.domain.com 1 1,3-5,8 config 
Unit Port: Status: Bol trlrer «Part. trap. Link trap. “Lank pulses DUD action 


ttt i as Ee ee ee ee eT eT errr ke ee ee ee 


1 1 on mac enabled 
I 3 on mac enabled 
at 4 on mac enabled 
1 3 on mac enabled 
1 6 on mac enabled 
1 8 on mac enabled 


cn nie Tei cir tenn an eee ee a 


disabled enabled N/A 
disabled enabled N/A 
disabled enabled N/A 
disabled enabled N/A 
disabled enabled N/A 
disabled enabled N/A 


Figure 1: Verifying port configuration. 
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RMON Stats Variable: Packets Received 
3 4 
Sg a a a rr i 
1 302414796 0 
2 0 34651342 
Help: ? 
Host: host.domain.com 


absolute 
0. 5/sec Sample: 57 


Figure 2: Dynamically monitoring RMON variables 
across all ports. 


Mode: 
Rate: 


Scenario 4: A recent addition to ND has been 
the ability to continuously monitor a specific MIB 
variable across all units and ports of a network device. 
Python includes a Curses module that allows for sim- 
ple screen control. This has been utilized to display a 
matrix of text where the columns represent slots or 
units, and rows represent ports. Figure 2 contains a 
single screen snapshot showing the RMON etherstats 
variable for the number of packets received. ND will 
query the device and update the screen every two sec- 
onds. 


Scenario 5: There are over 500 3COM FMS 
type devices at TAMU. Our monitoring strategy is to 
have statistics that are summarized on a weekly basis. 
Clearing all the statistics on every FMS then becomes 
necessary. ND was specifically designed to be 
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incorporated into shell scripts so that fairly complex 
tasks can be accomplished easily. Figure 3 shows how 
all the statistics on all FMS devices on campus can be 
cleared at the same time. 


( 


echo "fms ports" 
grep -v *~[#%]’ SFMSHOSTS | 
while read host 


do 
echo: “clear Shost Jot all Statistics" 
done 
echo “exit 
) | nd 


Figure 3: Shell programming with ND. 


Scenario 6: The characteristics of all ports and 
units for a given device can be shown in a single table. 
This can be very time consuming when logging 
directly into a network device. Also, the device’s soft- 
ware may not support tabular output of information. 
The ND command and resulting output for this exam- 
ple are shown in Figure 7. 


Scenario 7: With over a thousand network 
devices at TAMU, new devices are constantly being 
added and old ones replaced. Using ND, a device can 
easily be configured to a known state. Figure 4 
demonstrates how a set of specific ports can be config- 
ured easily using a shell script. 


nd-fms-ports> show fms-device-1.domain.com 1 all counters | \ 
awk: ae (ie CS). OO) print. oO) 
Uni Mult Bcast Tot Ucast Mcast Bcase LO tat 
Una “Bort Writs “trims! “remss siome ferme, “OCTS sOCTS: ~OCtS Octets Colls Runts 
i Z O% 1236 0 11 heed 0 0 0 116943 2 0 
at 3 0% 145846 Or 5 9.60< ae 0 0 O 20298744 209 0 
1 6 O% LO36 0 764 2402 0 0 0 468142 0 0 
1 i O% 6697 OQ: 200! LOOS 0 0 0 531474 105 0 
1 8 O% “1621.7 0 ooe 16865 0 0 0 1220390 16 0 
i) 10 O% 3268 0 18 3286 0 0 0 ZIONS A, 0 
Figure 5: Discovering which ports on a device are active. 
nd-fms-ports> show fms-device-1.domain.com 1 1-10 counters | sort -k 4 -r -n 
Uni Mult Bcast Tot Ucast Mcast Bcase Total 
Unit Port Util fems frms frms frms octs octs_ octs Octets Colls Runts 
1 3 0% 205380 O° 8453 213833 0 0 0: 31735634 666 0 
1 8 0% 23148 0 951, 924099 0 0 0 1726534 63 0 
1 7 0% 9413 O 1415 10828 0 0 0 746284 39) 0 
1 10 0% 4627 0 18 4645 0 0 0 324124 13 0 
1 6 0% ZO18 O° 1090 3608 0 0 0 679204 184 0 
1 Z 0% 1803 0 Ly 1820 0 0 0 164001 6 0 
1 9 0% 0 0 0 0 0 0 0 0 0 0 
1 5 0% 0 0 0 0 0 0 0 0 0 0 
1 f 0% 0 0 0 0 0 0 0 0 0 0 
1 1 O% 0 0 0 0 0 0 0 0 0 0 
Figure 6: Finding which ports on a device are using the most bandwidth. 
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connections in the building. Figure 8 shows the com- 
mands used to answer this query. 


Scenario 8: A customer calls and wants to know 
the speed of the workstation connections in his office. 
He provides the building and room number. The caller 
also wants to know the location of the 100 MB 


##!/bin/sh 


Scenario 9: A student in a residence hall has vio- 
lated the acceptable use policy and is causing a 


4 Find out which ports to configure as secure 


snmptable -mPRODUCTMIB -R STARGET public -H \ 
mrmPortTable mrmPortCardIndex mrmPortIndex mrmPortInterfaceType 
grep -twietedParr’ || sede “aio VC LOr OA). Choco ay) Nd 2 ee Ol 

( 

echo set read commread 

echo set write commwrite 

echo fms security 

while read PORT 


do 
echo set STARGET SPORT addresses 1 
echo set ! SPORT intrusion noAction 
echo set ! SPORT ntk NTKWithBcastAndMcast 
echo set | SPORT mode continuousLearning 
done 


echo quit 
echo exit 
) | nd > sdeys null 


Figure 4: Shell script for configuring a new device to a known state. 


nd-fms-ports> show fms-device-1.domain.com 1 all diagnostics 


Total Colls/ Runts/ Late 
Unit Port frms Coll’s Runts frame frame Frags colls 
ik 1 0 0 0 - : 0 0 
ik 2 2700 12 O 0.44% 0.00% 0 0 
i 3 273442 916 OF 3033. “Ost 0 0 
ik 4 0 0 0 . . 0 0 
1 5 0 0 0 . . 0 0 
i} 6 4656 283 O 6.08% 0.00% 0 0 
1 7 13857 620 O 4.47% 0.00% 0 0 
i 8 30582 93 0. 0.30%) “OF 00% 0 0 
at 9 0 0 0 : . 0 0 
i 10 5947 Zo O 0.39% 0.00% 0 0 
1 el tales 8 O 0.98% 0.00% 0 0 
ih 12 8095 42 O 0.52% 0.00% 0 0 
1 3 56349 2548 O 4.52% 0.00% 0 0 
1 14 5 0 O 0.00% 0.00% 0 0 
i; ie 0 0 0 - ; 0 0 
1 16 1574 ot 0 0.25% (0:.00% 0 0 
i i 30788 48 Or SORT 6%:. 05:0'10% 0 0 
1 18 M222 66 O 5.40% 0.00% 0 0 
1 19 0 0 0 : 7 0 0 
1 20 790 0 O 0.00% 0.00% 0 0 
1 al ZA 0 O 0.00% 0.00% 0 0 
i 22 0 0 0 : : 0 0 
i 2S 2805 40 O 1.43% 0.00% 0 0 
i; 24 0 0 0 - ‘ 0 0 
1 25 14047788 1032746 On F7soee “OEO0G 0 0 
1 26 0 0 0 . . 0 0 
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Figure 7: Displaying diagnostics for all units and ports in one command. 
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disruption. An engineer can disable the data ports to 
the room and schedule a meeting with the owner to 
discuss the problem. Figure 9 shows the commands 
used to solve this problem. 


Benefits 


One of the most important benefits that has 
resulted from the development and deployment of ND 
is the availability of a command line interface that is 
the same regardless of which type and make of net- 
work equipment is being accessed. This has always 
been the potential embodied in standard MIBs, though 
most network management application software has 
never taken advantage of it outside of an operational 
setting. The result has been that it is easier to train 
users. A module’s commands will be effective on 
numerous devices regardless of the type and make of 
the device. 


Another benefit that engineers have expressed is 
the ease of accessing ND and the simplicity of its 
interface. ND can be invoked from any Telnet/SSH 
session and is a system wide tool. No software needs 
to be installed in a user’s account or on a workstation 
in the office or in the field. Also, the online help and 
commonality of the syntax between modules makes it 
easy to remember commands and learn new modules 
quickly. Command line history files and command 
line editing also make it simple to look up recent com- 
mands or use a previous command as the starting point 
for a new one. The result is that engineers are very 
efficient when using ND. 


One of the seemingly mundane features but con- 
sidered very valuable by the operators and engineers 
at TAMU 1s the ability to view a condensed, full page 
of statistics with a single command. This makes it 
simple for operators and engineers to get a picture of 
the overall health of a network device and quickly 
identify a problem. Usually, this is an action that 
would require a cumbersome series of queries when 
using the vendor’s user interface. 


nd-netdb-drops> show room BLDG 311 


Drop number Type Room Length DB loss Dept. Media 


 —-—_<--——-— <-— —_ «+ —-— — — —e— -— *-—*e*-—— = —  oOOrlrhlUcr rhc Or Chr rh rhc rhc Or ri nr wr wr rr er rh rh rh rhc Orhlhcr Or rhc Or CO wr wr Hr wr wr wr wr nr wr nr rr rl rhc Or rChc rhc Or Or er re 


BLDG:1164 
BLDG? LOLS 


CAS 31 106 O20 
GATS: 3d 0 O50 


nd-netdb-drops> show device devl-ost5-1 


Device Unit Port Drop number Type 


devl-ost5-1l 4 1 BLDG:1136 CAT5 

deVvil--oSst5 = 1 4 2. .BLDGLLI37 CATS5 

devl=osta= 1 4 3 BLDG:994 CAT5 

devl-ost5-1l 4 a “BLpGid 107 CAT5 

devl-ost5-1l 4 5S- BLUG his CAT5 
4 6 


devl-ost5-1l BLDG:1064 GATS 
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The integration of the Fiber Circuits and Twisted 
Pair Drops database functionality has proven to be 
beneficial also. Leveraging the investment of learning 
ND and tying the information back to ND commands 
where appropriate not only saves human resources but 
has made the databases more valuable than they would 
Otherwise be. In many _ environments, network 
databases of these types are considered a burden and 
are many times discarded because they have a differ- 
ent user interface from other tools and because the 
data is trapped in a separate system and not available 
in other contexts. 


Another important benefit of ND is its ability to 
pipe output to a shell command or to a file. This fea- 
ture gives users a powerful means of extending the 
functionality of ND in the traditional Unix style. 
Unlike many network management applications, the 
designers of ND realized that they could not foresee 
all the desires that operators and engineers would have 
and made ND extensible in this way. This has saved 
users from having to develop new special purpose 
applications and again saved resources. 


Conclusion 


ND was designed to be a powerful network man- 
agement tool and to provide user-friendly functionality 
and extensibility. The Python language facilitates 
maintainable modules and a flexible command-line 
interface allowing the simple addition of new vendor- 
specific and standard MIB functionality as the net- 
work grows. The ND CLI eliminates the need for 
operators and engineers to learn complex vendor-spe- 
cific syntax by providing a new interface based on the 
standard MIBs implemented by all vendors. Unix 
authentication and authorization mechanisms provide 
a basis for managing access to network devices inde- 
pendently of the individual device’s authentication and 
authorization mechanisms. 


It is clear that this type of approach to network 
management software is a powerful one. The 


The building and room number are entered. 


Gonnected to. Unit. Port 


DEPT<1O0Baselx. ~devi-ost5s=1 4 7 
10BaseTX 


dev2-fms2-1 3 6 


One drop is IOMB, the other is 1OOMB. Check the 1OOMB 
device and see in which rooms the drops are terminated. 


—_——* = —_—*-*— = = = = — el erhlhc er er ler rhUcUhoO— + © 


147 O0 DEPT 100BaseTX 

eSe/, O30 DEPT? 100BaselTx 

O1l1A 0 820 DEPT 100BaseTX 
O11A 0 O06 DEPT 100BaseTX 
91 O20 DEPT 100BaseTX 

O O20 DEPT 1l100BaseTX 


Figure 8: Helping a user Locate 100 MB ports. 
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development of ND began from the desire to build a 
friendly application that was also a powerful tool. A 
simple command line interface leveraging Unix was 
key to accomplishing this. More importantly, ND 
became a platform tn which business practices could 
be embodied. At the same time, this embodiment can 
lead to weaknesses. Some business practices coded 
into ND may not be adoptable in other environments. 
They can even become a burden in TAMU’s environ- 
ment when it makes sense to change a business prac- 
tice but ND would require significant redevelopment 
to support the change. 


In the final analysis, every networking organiza- 
tion has a unique culture and a unique set of business 
practices. That organization must choose software that 
will support them in their mission. Tools that are too 


nd-ost-vports>? show devl-ost3-1 4 12 
SpLOt Port: otatus. VLAN MAC 


a ee ee ee 


TRNe detail au to 60 


4 12 on iM. SOO2.0 aaah b23 eZ 
nd-ost=vports> disable devi-ost3=1-4 10 
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low-level do not have interesting enough behavior for 
operators and most network management applications 
do not interest engineers because they are too restric- 
tive or because they are not adaptable to the organiza- 
tion’s business practices. Many times, the only per- 
ceived alternatives to this problem are to build com- 
pletely custom network management applications or to 
change business practices. At TAMU, ND has suc- 
cessfully proven that simply designing a better tool is 
a viable alternative with many benefits. 


Future Work 


We are currently looking at integrating ND with 
an event management package. The event manage- 
ment package could trigger a lookup of a network 
device at a specific event threshold, execute a set of 


The engineer checks the status of the port before disabling. 


Prot ae Mode Timer 


A st ttn rn Et se oo 


The engineer disables the port and is then 
required to document this action. 


Enter a comment to associate With this action. ianeident number 2000.143 


nd=Gst=vports> show devli-ost3-) 4 10 


Slot “Port Status. VEGAN MAC 


hs = = i re SS St eS ke i ir ir Te rt ee rt rr ST LT Se Ss et eS = 


Verify the status. Nowa "*’ reflects a comment is 
present for this port, and the status has changed to ‘off’. 


Encap Mode Timer 


- ep a tet zee ar aie ae ee ee See 


tes 0. =e hck i  O0820 aa bbs 2 32 TRNSdetauit® “auto 60 


nd-ost-vports> show devl-ost3-1 4 10 comment /fthe student telephones the Operations Center before anyone 


nd-ost-vports> enable devl-ost3-1 4 10 


Witace Ont 


tii nt Tt dn iI Es eee ee ee eS ae ee 


46o294 devi-ost3=)l-net-tamusedu. 4 10 


can reach and discuss the problem with him, the Operations 
Center can check the status and inform the student of the situation. 


Comment 


a st st tk st SS ss Ss eS 


incident number 607.689 


When the problem has been resolved, the port is re-activated. 


Enter a comment to associate with this: action: incident 60/7 .689 resolved 


nd-ost-vports> show devl-ost3-1 4 10 
plot Port. Status. VLAN MAC 


ee ee ee 


Prot Encap Mode Timer 


ss a en a eT 


Ae uO * On i O02 20 aa * bbe? 32 F2-TRN. sdetaudia: lato 60 


nd-ost-vports> show devl-ost3-1 4 10 comment The comments are still assigned and can be retained, or 


can be removed with the ‘comment delete’ command. 


Id Host Unit Port Comment 

46294 devl-ost3-l.net.tamu.edu 4 10 incident number 607.689 

46295 devl-ost3-1l.net.tamu.edu 4 10 incident 607.689 resolved 
Timestamp and audit information can be viewed using the 
‘comment info’ commana: 

as el iP sAdduesse: 192) .)68.. dual Users Operator 41 at server 

Device: devi=tms=) Unit I POET 26 

Date: 1999-10-19 Comment: disabled port for usage violations 


nd-ost-vports> comment delete 46294-46295 


Delete the comments by comment ID number. 


Figure 9: Turing off a student’s internet access. 
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predetermined ND commands, and generate a report 
for an operator to review and make a diagnosis. Addi- 
tionally, while the Fiber Circuits and Twisted Pair 
Ports databases have a good command structure for 
querying them, it can be cumbersome populating the 
databases with bulk data. Additional work needs to be 
done on simplifying and bulletproofing this process. 
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Combining Cisco NetFlow Exports with 
Relational Database Technology for 
Usage Statistics, Intrusion Detection, 
and Network Forensics 


John-Paul Navarro, Bill Nickless, & Linda Winkler — Argonne National Laboratory 
ABSTRACT 


Argonne National Laboratory operates a complex internal network with a large number of 
external network peerings. A requirement of this network is that it be monitored with minimal 
impact on traffic. Cisco NetFlow technology provides the information necessary to monitor such a 
network, but the data from NetFlow must be captured and analyzed. We present a system that uses 
a high-powered relational database to manage the data. Our primary motivations in building this 
system were to leam whether or not database technology was an appropriate tool for this situation 
and to understand what types of questions about the network could be answered with such a 
system. 


This work was supported by the Mathematical, Information, and Computational Sciences 
Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department 


of Energy, under Contract W-31-109-Eng-38. 


The Problem 
High Performance Network with Minimal Firewall 


Argonne National Laboratory peers with more 
than 50 external Internet Service Providers and Inter- 
net2 networks, at rates up to OC-12 (622 million bits 
per second). Soon we expect those peerings to 
increase in speed to Gigabit Ethernet and higher. 
Argonne’s networks are based on Asynchronous 
Transfer Mode (ATM) and switched 10/100/1000 
Megabit/second Ethemet. Argonne’s network has two 
separate border routers that handle these peerings. 
While these border routers can take over for each 
other in the case of a failure, normally they do not see 
each other’s traffic. 










Border 
Router 
1 


Border routers establish 
extemal connectivily 
with over 50 other networks 


Border 
Router 
2 





This is an interesting situation for a number of 
reasons. First, there is no single point in our network 
that can be used to monitor all traffic into and out of 
the lab. Second, the external network speed is a criti- 
cal issue to a number of research groups in the lab — 
slowing down the networks by running them through 
filtering or monitoring routers can have a huge impact 
on many experiments. And finally, the number of 
peerings and their exact topology frequently change as 
we modify our external networking relationships. 


What’s Going On in the Network? 


We needed a way to examine network utilization 
statistics, perform basic intrusion detection, and look 
at traffic pattems. The obvious way to do this would 


Sprni ATM Service 
















Figure 1: Router map. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


285 


Combining Cisco NetFlow Exports... 


be to drive all traffic through a router and use the 
router to watch every packet, but we are unwilling to 
crcatc an artificial bottleneck in our network for net- 
work statistics gathering or intrusion detection pur- 
poses. 


Unfortunately, many available network statistic 
gathering and intrusion detection systems, such as 
Network Flight Recorder, require that the intrusion 
detection system be able to inspect each packet as it 
enters and leaves the network. The Argonne network 
is not amenable to general packet inspection even at 
some small number of points for the reasons outlined 
above. 


Thus we needed to find some way to keep an eye 
on our network traffic without measurably impacting 
its performance. 


The Scenario 


Fortunately, we had other options to explore. 


Cisco has a technology known as “Netflow”, 
which provides a summary of traffic through a router. 
We also have some significant experience with 
databases, which led us to believe that a database 
would be the optimal way of tracking Netflow data 
from multiple routers. Initially, we weren’t sure how 
much data would be generated from our network, but, 
since we operate a number of supercomputers to sup- 
port scientific experiments, we felt that we probably 
had the computing resources on hand to at least study 
the situation. 


The motivating question for us was this: Is 
database technology a good way of tracking and ana- 
lyzing NetFlow data? 


Cisco Netflow Overview 


Here is how Cisco describes this technology: 
A network flow is defined as a unidirec- 
tional sequence of packets between given 
source and destination endpoints. Network 
flows are highly granular; flow endpoints 
are identified both by IP address as well as 
by transport layer application port numbers. 
NetFlow also utilizes the IP Protocol type, 
Type of Service (ToS) and the input inter- 
face identifier to uniquely identify flows 
[1]. 

The IP routers in our network do NetFlow 
switching already, as it is a very efficient way of 
enforcing Access Control Lists (ACLs). The first 
packet of a flow will be inspected to see if it should be 
permitted or denied. Once that determination 1s made, 
a flow record is placed in the router forwarding cache. 
Following packets are matched against the flow record 
and are forwarded appropriately. Once the flow record 
cache line times out, a UDP/IP packet is generated and 
forwarded out of the router to a collection station. 


The resulting NetFlow records provide a sum- 
mary of just about anything a network statistics 
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collector or intrusion detection system may want to 
know about the traffic, with the exception of the actual 
packet contents themselves. 


Database Technology 


Werun anumber of Oracle and MySQL database 
servers to support our researchers. One of the authors 
(Navarro) was a professional Database Administrator 
(DBA) for a large Oracle installation at a previous 
employer. 


Over the past several years, we have moved 
many of our critical infrastructure applications such as 
our list of hosts, lists of users, and so on, from flat 
files or simple lists into relational databases. Thus, 
many of our staff are becoming experienced with 
using databases for systems and network administra- 
tion. 


Thus we were fairly confident that a database 
would be the right solution for storing NetFlow data 
and that we would have the ability to manipulate it as 
we wished. The remaining question was — what should 
we use for a database server? 


High Powered Database Server 


We support a collection of workstations, clusters, 
and supercomputers as well as our networks. Our 
existing database servers run primarily on small Linux 
boxes or Solaris machines, and we knew that none of 
them would be up to the task of keeping up with the 
NetFlow data. The only spare computers we had were 
obviously underpowered. 


One of the supercomputers that we support is a 
96-processor SGI Origin 2000 [2] with approximately 
2 terabytes of Fibre Channel disk. This machine is 
primarily used to support computer science and com- 
putational science. So, we decided to run this project 
as an experiment and, therefore, to justifiably use the 
Origin 2000 to run the database. We were fairly confi- 
dent that it would have sufficient computing power to 
keep up with the NetFlow data during our initial tests, 
after which we could more accurately predict what 
kind of system would be necessary for production. 


The Creation 


Here, we describe the system that we built to 
capture and analyze the router NetFlow data. 


Netflow to Database on Origin 2000 


There are three basic parts to our configuration: 
the actual routers generating NetFlow data, a Perl [3] 
script to catch the NetFlow data from the network, and 
the back-end SQL database running on the Origin 
2000. We experimented with Oracle 81 [4] and 
MySQL [5] back-end SQL software. We used on-line 
data capture where incoming NetFlow data went into 
the database directly. Our Perl script that caught the 
NetFlow data from the network would simply make 
DBI [6] calls to insert the data into the back-end SQL 
database. The primary advantage of this method 1s that 
the database is updated in real time, but has the 
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disadvantage that data can be lost if the back-end SQL 
database is unavailable for any reason. 


We also used off-line data capture where the 
Incoming NetFlow data was written to a temporary 
disk text file and then periodically loaded in bulk to 
the back-end SQL database. This allowed us to con- 
tinue capturing data while the back-end SQL database 
was unavailable. It also allowed us to insert the data 
into multiple back-end SQL database engines when 
we wanted to compare their performance. 


Database Schema Design 


We chose a very simple database schema, drawn 
directly from the NetFlow specification itself. Our 
database consists primarily of a single table. Each 
row in the table is a single NetFlow record. Each col- 
umn in the table has a one-to-one correspondence with 
NetFlow fields. We added only one extra column in 
the table to identify which border router generated the 
flow record. 


create table netflows ({ 


FOUter ad ichar(l) not nul, 
SrC apn bigint unsigned not null, 
dst_ipn bigint unsigned not null, 
nxt_ipn bigint unsigned not nul, 
fin smallint unsigned not null, 
ifout smallint unsigned not null, 
packets integer unsigned not null, 
octets integer unsigned not null, 
Starttime timestamp nor null; 
endtime timestamp nov nul, 
srcport smallint unsigned not null, 
dstport smallint unsigned not null, 
tcp Ciny nit unsigned not null, 
prot tinyint unsigned not null, 
tos tinyint Gnsigned not nulds, 
srcas smallint unsigned not null, 
dstas smallint unsigned not null, 
srcmask tainyint unsigned not null, 
dstmask tinyint unsigned not null 
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This schema is so simple because we did not want to 
hinder our ability to make queries across the data later. 
We were interested to see how far modern SQL 
database engines could take us without optimizing the 
tables for particular types of queries. 


Using The System 


The system as described above is actually the 
complete system that we are using at present. We cre- 
ated the infrastructure to get the data from the routers 
into the database and then left it at that. Future steps 
(as discussed below) might be to provide nice inter- 
faces to the data in the database, but this wasn’t our 
initial goal. Rather, we wanted to see what kinds of 
questions could be answered with the data. From this 
point on, we’ve experimented with the system by 
forming basic SQL queries and examining the results. 
Our simple database scheme was designed with 
exactly this usage in mind. 


Once you have the NetFlow records stored in the 
relational database, you can run many different types 
of queries against them. Here, we present some of the 
ad-hoc queries that we’ve found that we frequently 
use. 


Usage Statistics 


This is an example SQL query for someone inter- 
ested in network statistics, perhaps asking the question 
“‘what Autonomous Systems (ASes) have we 
exchanged the most amount of traffic with recently?” 

i Give-a list of interesting 
## ASes (high traffic flow) 
select srcas,dstas,sum(octets) as bytes 
from netflows 
group by srcas,dstas 
having bytes > 10000000 
order by bytes 


The resultant table lists each pair of Source AS and 
Destination AS that we have exchanged traffic with, 


SQL 
Database 
Back-End 


e.g. Oracle, 
MySQL 
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Figure 2: Database layout. 
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including our own, with more than 10 million bytes 
throughout the data stored in the NetFlows table, with 
the Source AS/Destination AS pairs that exchange the 
most data at the top. 


Intrusion Detection 


A network security officer might be interested in 
what outside IP addresses are trying to perform scans. 
Here is a query that illuminates the answer to that 
question: 

Select sre ipn,count (distinct dst.4pn) 
as num_anl_addrs 
from netflows 

where srcas>O and starttime > 

date_sub(now(),interval 3 day) 
group by src_ipn 

having num_anl_addrs > 64 

order by num_anl_addrs 


This query gives the network security officer a list of 
IP addresses from extemal ASes that have contacted a 
large number of internal IP addresses. Given an IP 
address high on this list, the security officer might 
want to see what that IP address has been doing: 


Select. 6re i pn mines tarteime) cas first: 
max(endtime) as last, 
Count Cdistinet dst. ipn)as 
addrs;prot,a¢stports 
from netflows 
where src_ipn=140221009006 
group Dy protjdstport 
order by addrs 


ICMP Echo Reply (ping) scans, port scans, and even 
NMAP stealth scans show up very obviously in this 
and the preceding report. 


Network Forensics 


Let’s say that the network security officer sus- 
pects that a machine on her network has been compro- 
mised. The officer would like to go back in time to see 
where any network connections came from, what kind 
of network protocols were used to compromise the 
host, and see where any outgoing connections may 
have gone. The officer may even wish to look at other 
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machines on the same subnet. Here is one query that 
might be used to do this; see Listing 1. The officer 
then gets a list of all the traffic entering or leaving a 
network between two given points in time. Such a 
report can be used to narrow down the type of attack 
that might have been used, and whether the compro- 
mised machine(s) on that network were used to attack 
hosts elsewhere. 


Results and Practical Applications 


Sizing It: Performance and Resource Impact 


Not everyone has a spare 96-processor Origin 
2000 sitting around to host a NetFlow database. (Not 
even us, really.) Thus, one has to ask questions about 
the investment necessary in time, computer hardware, 
and software to appropriately host a NetFlow 
database. 


The first variable to consider is the rate of Net- 
Flow records generated by your routers. This rate will 
vary widely depending on your usage, whether your 
network 1s being actively scanned, and the time of day. 
The best way to get a handle on this question is to 
simply go ahead and install the Perl code that catches 
NetFlow packets and saves them as flat files. Run this 
for a few days, preferably during the workweek as 
well as over a weekend. You can then run simple line 
counting utilities over these flat files. 


Our routers generated about 14 million flow 
records on Monday, July 24 2000. Our Perl script 
stores the flow records in 5-minute batches. These 
batches ranged from 26,564 to 80,634 records per 
batch. 


It is important that your database engine can 
accept table inserts much faster than your routers can 
generate NetFlow records. You need some perfor- 
mance overhead to handle queries, outages, and time- 
outs. We recommend trying to choose a SQL back-end 
system that can handle at least triple your real-time 
NetFlow record generation. 


Using our example from July 24, we would need 
a SQL back-end database that can handle about 1500 


## Something very weird happening at a particular time on a particular net. 
select min(starttime) as -startt, max (endtime) as endt, 


Sre 1pm; GSt.ipm,. prot. Sreport, 
sum(packets) as pkts 
from netflows 
where ( ( starttime > 20000105003200 


and starttime < 20000105003600 ) 


or { endtime > 20000105003200 
and endtime 
group Dy sre 2pn, dst 1pm prot -dstpore 


having ( src_ipn >= 146137000000 


and) Src pi <=" e637 2 S52 oor ae IO 


( dst_ipn >= 146137000000 and 
dst ipm <=" 146137255255 >) 


< 20000105003600 ) 


dstport; 


) 


order by pkts,Sreipn,dSt 29n,prot.,sreport-dst port 


Listing 1: Examining other machines on the same subnet. 
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Insertions per second at a bare minimum. Preferably 
we would want a database that could handle triple that 
rate, or just under 5000 insertions per second. 


The next variable to consider is how long you 
want to keep NetFlow records around. This dictates 
your overall storage requirements. Be aware that the 
size of your NetFlow table may impact the speed of 
table inserts, which feeds back to into the performance 
requirement discussed above. 


We chose to keep about two weeks’ worth of 
data on line. This time frame is a compromise between 
our desire to protect the privacy of our network users 
and having the data around to look into unusual 
events. To protect the long-term privacy of our net- 
work users we specifically chose not to back up the 
database itself. 


Turnaround Time vs. Insertion Performance 


You will also choose which table columns to 
index, if any. This is a tradeoff between insert perfor- 
mance and query performance. If you intend to simply 
capture data and look at it when something bad hap- 
pens, you may not wish to spend the cycles indexing 
data that will be thrown away. The downside is that 
every query will require a full scan through the data 
before it can retum results. 


Some queries just about require a full table scan, 
such as the Intrusion Detection example query we pro- 
vide above. If you anticipate doing those types of 
queries on a regular basis, you might choose to buy 
the fastest disk system you can find and not bother 
doing any indexing at all. 

Database Technology Choice Makes a Big Differ- 
ence 


As mentioned above, we tried MySQL and Ora- 
cle on the SQL database back-end host. As part of 
their license agreement, Oracle doesn’t let you publish 
specific performance numbers. But we can make some 
general comparisons between the two systems. 


selectisre-ipn,dst-i1pnsprotySreport, dst port, 
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Oracle costs money. MySQL is (pretty much) 
free. 


Oracle supports transactions, which slows it 
down but protects the database against corruption if 
the underlying host crashes for some reason. If the 
host system crashes under MySQL, you often have to 
re-index the table completely, which can take a very 
long time. 


The MySQL query optimizer is sometimes less 
sophisticated than we might wish. Consider the query 
in Listing 2, assuming that the src_ipn and dst_ipn 
columns are indexed. This query took about 8 min- 
utes to run, returning 576 records. The MySQL engine 
did a full 175,460,008-record table scan looking for 
matching rows. 

However, the slightly simpler query in Listing 3 
took 0.04 seconds to retum 16 records. This time the 
MySQL engine used the src_ipn index to find the 
matching records much more efficiently than doing a 
full table scan. (The other obvious query using the 
dst_ipn index took about five seconds and returned the 
other 560 rows.) A more sophisticated query optimizer 
than MySQL provides would have split the original 
query into the two much simpler-and faster-queries 
and then combined the results. 


If you’ve already decided to use MySQL, obvi- 
ously you will want to run the two fast queries instead 
of the one slow query. But if you haven’t yet decided 
on a SQL database back-end system, you may wish to 
run some sample queries and see how the optimizer 
treats each of them. The query optimizer is also criti- 
cal to achieving good performance in a SQL database 
that supports parallel queries. 


Other database products may provide features 
that could be used to speed up certain queries. Con- 
sider the Usage Statistics example query. If one were 
to drop the time constraint, it might be a perfect fit for 
an IBM DB2 [7] Summary Table. That is, the statistics 
would be generated “‘on the fly” during the NetFlow 
record insert process. 


sum(octets) as bytes, min(starttime) as first, 


max (endtime) as last 
from nettlows 


where sre apn=146139112126-0r dist +1 pn—-146139112126 
proup: by: “Src. ipnydst_1pn. prot, sreportdstport 


order by bytes 


Listing 2: One form of query. 


Select sTre-1pn dst ip, protasreport,dstport. 


sum (octets) vas bytes, min(starttime) as first, 


max(endtime) as last 
from netflows 
where src_ipn=146139112126 


group by Src A pn ,dst_ipn prot. sreport -dastport 


order by bytes 


Listing 3: Quicker query. 
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Futures and Conclusions 


Non-Supercomputer Database Hosts 


We cannot dedicate our 96-processor Origin 
2000 to this application. So we must choose the most 
efficient combination of disk drive(s), CPU, memory, 
and database software to meet our NetFlow record 
generation rate and query types. 


We may even choose to run several different 
SQL database back end servers, each optimized for a 
particular type of query. 


We are still in the process of deciding exactly 
what system to use for this. As usual, it comes down 
to more of a question of budget than necessary tech- 
nology. 


Higher Level Tools Necessary For Non-DBAs 


We are fortunate to be a cross-disciplinary team 
of network professionals, system administrators, and 
database administrators. Each of us is comfortable 
with writing simple ad-hoc SQL queries to look at the 
NetFlow data. Over time we hope to build higher-level 
tools that could be accessible to a wider audience. 
Examples might include a web interface that network 
security officers could use to view the activities of a 
given host, email alerts based on intrusion detection 
queries, or even graphics of network utilization over 
time. 


Database Technology Is a Good Solution 


The good news is that the higher-level tools can 
be built using the wealth of tools available for SQL 
databases. There is any number of commercial and/or 
Open-source systems available to make web pages that 
use SQL database back ends. Just about any report- 
writing program can take data from a SQL query. 


Database technology provides a good abstraction 
for this problem. It separates the nitty-gritty details of 
data storage from the actual problem we are interested 
in solving. As database software technology improves 
we get the benefits of those improvements right away. 
We are not stuck with a particular architecture tuning 
that limits us from following the Moore’s Law 
increases in processor and disk storage performance. 
As our needs increase, we can swap out an under-per- 
forming SQL back end system entirely, as needs and 
budgets permit. Finally, we can take advantage of all 
the data protection and administrative expertise of an 
existing SQL infrastructure, 


Availability 


All of the code we used in this work is written as 
Perl and Bourne Shell scripts. There are probably no 
more than about 200 lines of actual source code. Look 
for a pointer to the NetFlow Database Perl Code on 
the http://www.mcs.anl.gov/systems/software/ web 
page by the time this paper 1s presented at LISA. 
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ABSTRACT 


Many Cisco routers and switches support NetFlow services which provides a detailed source 
of data about network traffic. The Office of Information Technology Enterprise Networking 
Services group (OIT/ENS) at The Ohio State University (OSU) has written a suite of tools called 
flow-tools to record, filter, print and analyze flow logs derived from exports of NetFlow 
accounting records. We use the flow logs for general network planning, performance monitoring, 
usage based billing, and many security related tasks including incident response and intrusion 
detection. This paper describes what the flow logs contain, the tools we have written to store and 
process these logs, and discusses how we have used the logs and the tools to perform network 
management and security functions at OSU. We also discuss some related projects and our future 


plans at the end of the paper. 


NetFlow Accounting Records 


We should start with a more complete descrip- 

tion of what the flows are. Quoting from Cisco: 
A network flow is defined as a unidirectional 
sequence of packets between given source and 
destination endpoints. Network flows are 
highly granular; flow endpoints are identified 
both by IP address as well as by transport layer 
application port numbers. NetFlow also utilizes 
the IP Protocol type, Type of Service (ToS) and 
the input interface identifier to uniquely iden- 
tify flows [3]. 


A NetFlow record is created when traffic is first 
seen by a Cisco router or switch that is configured for 
NetFlow services. Flows are identified uniquely by 
characteristics of the traffic that they represent, includ- 
ing the source and destination Intemet Protocol (IP) 
address, IP type, source and destination Transmission 
Control Protocol (TCP) or User Datagram Protocol 
(UDP) ports, type of service and a few other items. 
NetFlow records end and are sent to the logging host 
on at least the following conditions: 

e For flows representing TCP traffic, when the 
connection is done (after a RST or FIN 1s seen) 

e When no traffic for the flow has been seen in 
15 seconds. 

e 30 minutes after the start of the flow. This 
causes long lasting traffic pattems to show up 
sooner than they might otherwise in the log. 

e When the flow table fills. 


Each NetFlow record contains data about the 
packets that are represented in that flow in addition to 
the unique identifiers listed above. These data include 
the start and end times for the flow, the number of 
packets and octets in the flow, the source and destina- 
tion Autonomous System (AS) numbers, the input and 
output interface numbers for the device where the Net- 
Flow record was created, the source and destination 
net masks and, for flows of TCP traffic, a logical ‘or’ 
of all of the TCP header flags seen (except for the 
ACK flag). In the case of Internet Control Message 
Protocol (ICMP) traffic, the ICMP type and subtype 
are recorded in the destination port field of the Net- 
Flow records. 


For example, suppose that a SSH connection is 
established from a client on host 128.146.222.233 port 
1234 to a server on host 131.187.253.67 port 22, and 
that the traffic passes through a Cisco device that has 
NetFlow processing enabled. We will simplify things 
and identify our flows here by a tuple containing the 
IP Protocol type, source IP address, source TCP/UDP 
port, destination IP and destination TCP/UDP port. 
The initial packet from the client to the server causes 
the router to create a flow entry for {TCP, 
128.146.222.233, 1234, 131.187.253.67, 23}. The 
response from the server to the client causes the router 
to create a related flow {TCP, 131.187.253.67, 23, 
128.146.222.233, 1234}. Data from subsequent traffic 
will be aggregated in these two flow records until one 
of the ending conditions listed above is seen, such as 


tea show, ipscache-ikels N87 259 407, Woe 2 0942 99. Oo bow 


SrCLeE SrclPaddress Doere DstIPaddress Pe orer DSte “Pts 
ADITO se 2S 1G 293 ATS On | 134). 1672.253:.:67 “06 03FA 0016 4 
AT3/ 0.1 IBts Bi 2259-67" ATZ/O0.30 210610672729 93) 06 0016.0ln4 8 


Figure 1: Active flows as seen on a Cisco router. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 291 


The OSU Flow-tools Package and Cisco NetFlow Logs 


when the TCP session ends, or because there has been 
no traffic for 15 seconds. Active flows can be viewed 
in the router command line interface with the com- 
mand show ip cache <IP Mask> flow. This allows you to 
view flows that exist on the router whose NetFlow 
records have not been exported yet (Figure 1). 


In the simplest case for a TCP session there will 
be a single flow representing the traffic from the client 
to the server, and a single flow representing traffic 
from the server to the client. The TCP flags field for 
both flows would typically have both the SYN and 
FIN bits set, indicating that packets with those flags 
had been seen traveling in both directions. 


This is not typical, however. Traffic for a single 
TCP connection is frequently represented by multiple 
flow records, due to timeouts from lulls in the conver- 
sation, the flow table filling up, or the 30 minute flow 
maximum lifetime. This means that one often has to 
string multiple flow records together to get all of the 
data corresponding to an entire TCP session. In these 
cases, the TCP flags field can be used to determine 
whether a flow represents data from the start, middle 
or end of the TCP session. Flows fiom the start of a 
session will have the SYN (but not FIN or RST) bit 
set, flows from the middle of the session will typically 
have no flag bits set, and flows from the end of the 
session will have the FIN or RST bits set (but not 
SYN). 


Flows for UDP and ICMP traffic behave simi- 
larly, although it 1s important to note that since neither 
of these are connection oriented protocols flows of 
UDP and ICMP traffic are just collections of similar 
packets. 


Terminology 


A NetFlow device is a Cisco router or switch 
that supports NetFlow services and which is exporting 
NetFlow records. NetFlow Protocol Data Units 
(PDUs, also called NetFlow records) are the account- 
ing records that NetFlow devices emit. We will use the 
term flow records or flow logs to refer to flow 
accounting records in the internal flow-tools format. A 
NetFlow collector is a host running flow-capture to 
create a flow log from NetFlow records exported from 
one or more NetFlow devices. Note that flows are un1- 
directional collections of similar packets. We will use 
the terms connection or session to refer to all of the 
packets associated with bidirectional communication 
between a client and a server. A connection (such as a 
connection from a telnet client to a telnet server) will 
consist of at least two and possibly many more unidi- 
rectional flows. 


The OSU Flow Tools 


The Ohio State University has written a suite of 
tools for collecting, filtering, printing and analyzing 
Cisco flows. The tools are written to work as UNIX 
pipelined commands, making it easy to perform data 
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reduction without creating unnecessary intermediate 
files. The tools are grouped roughly as “capture 
tools,” “general analysis tools,” and ‘‘security tools” 
in the following discussion. 


Work on flow-tools started in August of 1996 
when Cisco had released an EFT image with a new 
feature called NetFlow switching. At this time OSU 
had Internet connectivity via CICNet, and a local peer- 
ing with the state network, OARnet. We needed a way 
to determine how much of our traffic was staying local 
to CICNet, and indirectly the other big 10 schools, and 
how much was transiting CICNet to MCI. Where our 
traffic was terminating and originating would poten- 
tially influence future Internet bandwidth purchasing 
decisions. An initial release of flow-tools aided by 
testing and feedback from other CICNet member 
schools was generating statistics in early September. 
Since then features, fixes and documentation have 
been added to the tool set, primarily for OSU’s inter- 
nal use with incident response and traffic analysis. 
Cisco has improved NetFlow switching over the past 
few years including features such Border Gateway 
Protocol (BGP) AS information, aggregated flow 
exports, and integration with dCEF to provide what 
we are now using, NetFlow accounting. Much of the 
work OSU has done has been made publicly available 
in open source form. 


Internal Operation 


NetFlow records are exported from Cisco gear in 
one of several versions. To accommodate the slightly 
different records, the flow-tools package receives 
these records in native Cisco format and translates the 
records to a fixed size record stream format which 
contains a snapshot of what was available in the Cisco 
version 5 NetFlow export records. This conversion 
work is done in the programs that receive flow exports 
from devices (flow-capture and flow-receive), and 
most of the rest of the tools work with this internal 
flow-tools representation (flow-fanout 1s the exception 
— this is a generic UDP packet multiplexer). Most of 
the programs in the flow-tools suite were designed to 
read and write to stdin and stdout, although some have 
command line options that allow you to redirect I/O to 
specific files. The tools also support zlib compression 
(RFC 1950 [10]) on the fly to conserve space. 


Capture Tools 


Each NetFlow enabled router or switch has to be 
configured to export their flow records to a flow col- 
lector. Flows are exported through UDP packets sent 
to a designated host and port, where some sort of net- 
work service is expected to receive the data and do 
something useful with it. The IOS command jp flow- 
export destination 10.0.0.1 12345 would cause a device 
to export NetFlow records to the host with IP address 
10.0.0.1 at UDP port 12345. Several packages are 
available for receiving and processing these NetFlow 
exports, see the related work section at the end of this 
paper for a brief survey. 
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In the case of the flow-tools package this collec- 
tion service is provided through a program called 
flow-capture. Flow-capture listens on the designated 
UDP port and writes the received flow records to log 
files. To keep individual logs from growing to an 
unwieldy size flow-capture rotates to a new file peri- 
Odically. Flow-capture was designed to facilitate the 
long-term archival of flow records and has a built-in 
mechanism to manage the log file space in the output 
directory, either by restricting the number of separate 
log files that are maintained or by restricting the total 
bytes allocated to log files in the output directory. The 
UDP port, output directory, rotation period, and log 
count or size limits are all configured through the 
command line. For instance, flow-capture -E1G -n23 
-—p9991 -w/var/log/flows -z6 would cause flow-capture to 
listen on UDP port 9991, write its logs with level 6 
compression to /var/log/flows, rotate the file once per 
hour, and save up to one gigabyte of log files in the 
output directory. 


Flow-capture also allows you to connect to it via 
TCP to receive a real-time feed of flow records, which 
you can receive using the ne (netcat [8]) program, as 
in nc capturehost 9991 | flow-print. This can be useful for 
debugging as well as for applications that require real- 
time access to NetFlow records such as flow-dscan. 


There 1s also a simple program called flow- 
receive which listens for NetFlow records on a UDP 
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port (like flow-capture), but which writes the resulting 
flow records to stdout rather than archiving them to 
files. This 1s useful for debugging NetFlow exports. 


Flow-fanout captures NetFlow records through a 
UDP port (like flow-capture) and replicates them to 
multiple destinations, which are specified by host and 
UDP port on the command line. This is primarily use- 
ful for debugging purposes, though it can also be used 
to replicate flow records to other tools, such as 
cflowd. 


Flow-mirror and flow-rsync are simple scripts 
that copy flow logs from the collection hosts to the 
archive hosts using FTP and rsync, respectively. 


Flow-expire implements the same space man- 
agement features that flow-capture does. This is useful 
for managing the logs on systems that have copies of 
the flow logs mirrored through the flow-mirror or 
flow-rsync scripts but which are not running flow-cap- 
ture. 


Our architecture for flow collecting and process- 
ing has grown from a single Sparc 5 equipped with a 
few Gigabytes of disk space in August of 1996 to 
eleven Pentium and Pentium II based flow collectors, 
a dedicated Pentium III file server equipped with two 
250 gigabyte RAIDS arrays, and two one gigahertz 
Athlon processing servers. The flow collectors are 
typically connected directly to the routers they are 


Example Topology with Netflow enabled routers 
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Figure 2: Simplified diagram of the OSU network. 
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gathering flows from and also can double as reverse 
terminal servers and consoles for other physically 
adjacent equipment. These collectors have minimal 
local disk space, so our primary file server polls the 
flow collectors once an hour with rsync for completed 
flow exports. We use flow-capture to receive flows 
and manage the disk space on the collectors, and flow- 
expire to manage space on the file server. Figure 2 
shows a logical diagram of the OSU backbone and its 
connections to the Internet. Figure 3 gives an idea of 
how we manage flow captures. 


Flow collector performance depends directly on 
the availability of enough free CPU cycles to com- 
press the inbound flow exports and move the com- 
pressed stream to disk. To verify that the collector is 
not over subscribed, poll the kernel statistics for 
dropped UDP datagrams due to full socket buffers. On 
FreeBSD this can be done with netstat -s. Flow-cap- 
ture attempts to use larger than default socket buffers 
to help ensure bursty flow exports are handled without 
unnecessary packet loss. Flow-capture is started with 
rtprio which gives it an unfair advantage over other 
running processes. 


Disk IO and CPU both contribute to the perfor- 
mance of processing the flow data. Initially we would 
collect and process flows from the campus border on a 
single server. At that time most of our other routers 
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were not capable of generating flow statistics due to 
hardware constraints or were not able to run the Cisco 
images that supported the NetFlow feature set. Today, 
NetFlow exports are archived from all the campus 
backbone routers to a single large disk store which is 
shared with two other servers dedicated to processing 
the data sets and generating reports. Each of the 
servers where the data is crunched also have a striped 
two disk temporary storage area to reduce NFS traffic 
when iterating over the same data set many times, as 
we typically do. A simple benchmark of using flow- 
cat to uncompress a days worth of flow exports from a 
single router and print it in ASCII leads to a process- 
ing speed of about 300,000 flows per second on the 
one gigahertz Athlon boxes using local storage. Typi- 
cal pipelines such as flow-cat <dataset> | flow-filter | flow- 
stat <options> can reduce this down to around 245,000 
flows per second, which is adequate for the number of 
reports we run and the size of the data sets currently in 
use. Our upgrade plans include a faster RAID con- 
troller and a newer version of FreeBSD with improved 
NFS and network performance. 


We currently average 67,320 octets per flow and 
92 packets per flow. One of our busiest routers han- 
dles 397 gigabytes of traffic per day (about 35 
megabits per second) in 548 million packets which 
generates 5.9 million flow records per day. The flow- 


Distributed Netflow collection and processing 
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Figure 3: Simplified diagram of flow collectors at OSU. 
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tools flow record is 60 bytes long, so this works out to 
about 350 megabytes of flow logs per day at level 0 
compression. We use level 6 compression by default, 
where we get a 4.3:1 compression ratio, so this actu- 
ally works out to about 82 megabytes of actual disk 
space. This number can vary considerably — various 
denial of service attacks (especially SYN floods) can 
greatly increase the number of flows created, and 
accordingly, the number of flow records reported. 


General Analysis Tools 


Since our logs are split into relatively small files 
that represent network traffic for short time periods, 
we need a way to catenate these files together prior to 
processing. We can not simply use the UNIX cat pro- 
gram since each log starts with a general header that 
describes the contents of that log and this header needs 
to be removed from within the catenated files. The 
flow-cat program reads one or more flow logs (listed 
on the command line, though it will also read from 
stdin) and catenates the contents of these files in turn 
to stdout (or a designated output file). 


Flow logs are written as binary records to a file 
and are not directly readable. The flow-print program 
reads records from a flow log and prints their contents 
in one of several output formats. For example, flow-cat 
* | flow-print -f 5 would print information about each of 
the flows in all of the logs in the current directory 
using format 5. Format 5 includes the start and end 
time of the flow, source and destination IP address and 
TCP or UDP ports, IP protocol type, source and desti- 
nation interface numbers, TCP flags, and a count of 
the number of octets and packets for each flow. In the 
example in Figure 4 we have removed several of the 
output fields to make it more readable. The column 
labeled “‘p” is the IP protocol type — 6 is TCP, 17 is 
UDP. The column labeled “f”’ is the OR of the TCP 
flags for each flow. The last two columns, labeled “‘#” 
‘and “octets”? show the total number of packets and 
octets for each flow. We also shortened the timestamp 
format in the “start time’? column — normally times- 
tamps are printed as MMDD.HH:MM:5SS.SSS, so a 
timestamp of 0927.18:30:23.562 would represent the 
time 18:30:23 .562 on September 27. 


You can use command line options to cause 
flow-print to translate IP addresses and port numbers 
to names, where possible. We do not use this option 
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very often, since we are used to working with numeric 
IP addresses and port numbers. Also, the translation to 
host names can lead to misleading interpretations due 
to DNS spoofing (through cache poisoning or other 
techniques) and due to port overloading (running a 
web daemon on TCP port 23, which would be 
reported as a telnet connection). 


You can search flow logs and pull out interesting 
records with flow-filter. Flow-filter allows you to 
match records by the following fields: 

e Source or destination autonomous system num- 
ber (-a and -A options). 

e Source or destination port number (-p and -P 
options). 

e IP protocol type (-r option). 

e Source or destination IP addresses, using Cisco 
standard Access Control Lists (ACLs, -S and 

-D options). 

e Input or output interface numbers (-i and -I 
options). 

Most of these options allow you to specify 
ranges and lists of values to match. For instance, flow- 
filter -- 6 -P 21,23,25,80 would match TCP flows (IP 
protocol type 6) with a destination port of 21, 23, 25 
or 80 (which are usually FTP, telnet, SMTP and web 
services). The ability to filter flow records using Cisco 
standard ACLs allows us to perform powerful 
searches through our archives as part of incident 
investigations. The command flow-filter -f flow.acl -S 
attackers -D victims would read ACL definitions from a 
file named flow.acl (see Figure 5) and match flow 
records where the source IP address matches the 
“attackers” ACL and the destination IP address 
matches the “victims” ACL. You can also use the 
command line options to further filter the output by 
other fields, such as by IP protocol or TCP/UDP port. 


We wrote a related script named flow-search to 
facilitate investigations of computer intrusions using 
the flow logs. Flow-search allows you to easily apply 
the same filter to a large number of separate log files. 
You could easily do this by applying flow-filter to the 
output of flow-cat, but this procedure disassociates the 
flow records from the name of the log file that they 
were found in, which can be inconvenient in incident 
investigations. Flow-search applies flow-filter to each 
log file individually and keeps the results in derivative 
files named after the source logs. 


start Sic src. dst dst pf # octets 
time D port ap DOrt 

O02 003 a3 8.0" 164007. 1.2 O26: 2054136 .254:: 1.95 LO OOl Vly Os EE 36 
COS00 a5 34-2 1 6p. Nee 27 OS. loan TOy ess 2800U Sie SOs. oi 36 
OOZ00eT1 2384 640107. 1.3 28 00I 216 bo laoa2ed POSo+, 87-0) it 68 
COSOOn dT T2592) 6A Oy. lta 270522 4593 ook 23 LAO. ee sO 3) S129 
O020 O73 92 64a 107 wal P03 4) 2052 188.254.2097 £0007 [170° 1 48 
OOOO eso 2. 12 oe eer 17) Do) 206 sro ee a a3 Sys wk 61 
00:00:11 2404 204m 20241294230 SO" 140.2 54 4-..6 12d 6.3 30) be 79 


Figure 4: Sample output from flow-print, edited to make 1t more compact. 
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Since flow records are created when their corre- 
sponding flows end the records are recorded in the log 
file in order of the ending time of these flows. This 
makes it hard to correctly interpret the contents of the 
flow logs, For example, you may suspect that an 
intruder is logging into a host through a backdoor of 
some sort, and you can see network activity coming 
from the host (scans, exploit attempts, IRC connec- 
tions, etc) that lead you to believe that the intruder 1s 
active. But the flows associated with the backdoor 
connection may not show up in the flow logs until 30 
minutes after the backdoor traffic actually started, so 
the other activity may actually occur first in the log. 
We wrote flow-sort to quickly sort the flow logs into 
chronological order according to the starting time of 
each flow. Flow-sort is implemented as a rolling heap 
sort, where flows are sorted into a heap which repre- 
sents a 35 minute wide window (the maximum dura- 
tion of a flow is 30 minutes). As flows move out of the 
rolling 35 minute window they are written out, now 
sorted by the starting time of the flow. Flow-sort can 
also save the heap when it reaches the end of the input 
and can restore from a saved heap when it starts up on 
a new file, which allows us to correctly sort flows 
between different log files. 


Since flows are unidirectional and do not contain 
any indication about who initiated a connection, it can 
be difficult to correctly determine the client/server 
relationship between the source and destination hosts 
in each flow,. One can apply heuristics based on the 
transport level source and destination port numbers, or 
by maintaining state about previous network activity 
on each host (“‘host A created a connection to TCP 
port 21 on host B’’) and using that to make inferences 
about unknown traffic (“this traffic from {TCP, A, 
12345} to {TCP, B, 32145} might be a passive mode 
FTP data connection’’). These are subject to mislabel- 
ing (“‘this specific traffic was a backdoor to host B”’) 
or ambiguity (is traffic between TCP port 2000 and 
port 6000 is either OpenWindows, X, or something 
else entirely). The TCP flags field is of no help, since 
the same flags are usually seen in aggregates of pack- 
ets in both directions. It would be useful if flows of 
TCP traffic where a TCP packet contained a SYN but 
not an ACK were marked in some fashion, but Net- 
Flow services does not provide that information. 


! permit anything 
ip access-list standard all permit any 


matan. the attackers 
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We can try to use the starting times of the flows 
to infer the client/server status of the endpoints — the 
source of the first flow is the one that initiated the con- 
nection. Flow-connect attempts to make these infer- 
ences from sorted flow records and aggregates flow 
records for the same session into a single flow record 
whose source IP is the client and whose destination IP 
is the server. At the time of this writing, this is still 
something of a work in progress, though preliminary 
results are encouraging. 


Network Planning and Performance Tools 


We have depended primarily on other tools such 
as statscout [13], MRTG [14] and a variety of home- 
grown tools for network planning, performance anal y- 
sis and monitoring. We have found it useful to gener- 
ate some common reports from the flow logs. Flow- 
stat summarizes flows into useful reports, some of 
which are easily viewable as graphs using tools like 
gnuplot. Flow-stat currently supports almost 20 differ- 
ent report styles, including summaries by source or 
destination AS number, port, or IP address, and 
source/destination matrix summaries. See the section 
on network planning and performance tools for some 
examples of how we use flow-stat. 


Flow-profile is similar to flow-stat but provides 
the ability to aggregate flows based on groups of hosts 
as defined in a configuration file. This provides a 
powerful mechanism for additional data analysis, and 
is especially useful to generate usage data by group 
which can be used as a source for usage based billing. 
Flow-profile reads a configuration file which essen- 
tially describes the mapping between billing units and 
IP address ranges. The data from the flow records 
passed to flow-profile are aggregated according to 
these groupings and are summarized in the output, 
which is designed to serve as input to a billing system. 
See Figure 6 for an example of a configuration file, 
and Figure 7 for sample output. 


The flow-tools package is a small set of pro- 
grams for processing NetFlow exports that can be 
chained together, along with other standard utilities 
such as awk, grep, perl, sort, etc to produce reports. 
Leveraging off existing well-known utilities can lead 
to simple one-liners that produce detailed network 
activity reports or data sets for depicting long term 


ip access-list standard attackers permit 10.0.0.1 0.0.0.0 
ip access-list standard attackers permit 128.146.222.0 0.0.0.255 


ip access-list standard attackers deny any 


bP Match the: vietams 


ip access-list standard victims permit 140.254.1.1 
ip access-list standard victims permit 140.254.1.2 
ip access-list standard victims deny any 


O0070 
0202-020 


Figure 5: Example access control list definition file for flow-filter. 
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usage with utilities such as gnuplot or mtv. The fol- 
lowing are examples of some of the types of reports 
that can be generated. 


Top 10 Users 


It 1s easy to create custom reports. Here is how 
we find the list of the top 10 network bandwidth users 
in our dorm network, ResNet (see Figure 8). We use 
flow-cat to catenate the data sets for a day’s worth of 
logs together and pass the resulting output to flow-fil- 
ter to isolate traffic with an input interface of 1 and an 
output interface of 47 or 62 (our connections to the 
outside world, see Figure 2). The output fiom flow- 
filter is passed to flow-stat, which we will instruct to 


## define the inside interface 
inside: ) 43.4 


4 define outside interface 
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generate a usage report based on source IP address 
(-f9) and to report the totals in percent/total form. 
Finally, we filter out the comments with grep and sort 
the results in descending order by the value in the total 
octets column. If you examine the numbers you will 
see that the top 10 users are using almost 30% of the 
total octets counted. 


Counting Addresses 


First, lets count the number of unique IP 
addresses on the Internet that have exchanged traffic 
with hosts at ResNet. We will use flow-filter to pull 
out just traffic going to the Internet, and flow-stat for- 
mat 8 (statistics by destination IP address) to generate 


outside 8 

range 128.146.070.001 128. 146..070..001 math 

range 128.146.024.001 128.146.024.001 physics 

range 12824146.225.001 128421464225,001 economics 

range 128.146.216.001 128%. L464216"001 math 

range 128.146.222.001 128% 1462222001 athletics 

range 164.107.005.151 164.107.005.244 physics 

range 128.146.050.240 128.146.050.244 psychobotany 

## define these after ranges to make sure all addresses in a block 


## have range definitions 
bound, 2128.1-46:,0.0- 126. 146. 2554259 


Figure 6: Sample configuration file for flow-profile. 


#eroup octets packets 
+ in out dom out 
math 358627478 3423321 3274516064 4518383 
physics 7449413327 38512040 32410612445 44213679 
psychobotany 2025644653 27020722 14947199604 28384258 
english Deron 2283 2108750 2495 
Figure 7: Sample output from flow-profile. 
flow cat ier05..2000-09-=26 297 /|(Stlow=ta lier ad 21474620 || tlowastat 2) -£9) <\ 
bwrepiey = | sort sneer f= 3 

IB flows octets packets duration 

164510 7es/0.189 O77, To ATT 3.669 i053 

L400 q2 54229202 0.091 5036 2.487 On 71-9 

140 4254,..233.119 0.116 3.874 1. 936 0.824 

T6440 Jex67 4166 C2262 2.412 1.240 O25 30 

LAO 254. 106226, O..122 2.358 13.06 0.754 

140 92544.229.255 0.045 beo77 0.969 0.264 

164.107.86.143 0:70:59 Looe On 988 02837, 

164#107.93.206 0.091 602 On 857 0.390 

164.107.90.220 O2238 L390 0.742 Ox 352 

140 22544236.103 0.013 1.381 0.714 OR 152 
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the unique list, which we will count with the wc 
(Word Count) program: flow-cat cf05.2000-09-26.* | flow- 
filter -147,62 | flow-stat -f8] grep -v' # | we -L. 


From this, we see that there were 945,189 unique 
destination IP addresses contacted. 


Similarly, this will count unique source IP 
addresses at ResNet that have sent traffic to the Inter- 
net: flow-cat cf05.2000-09-26.* | flow-filter -i1 -147,62 | flow- 
stat -f9 | grep -v’ # | we -. 


This reports that there are 6,209 unique source IP 
addresses active. 


It might be interesting to count the unique IP 
addresses at ResNet that have received traffic from the 
Internet. We will turn the filter around (the destination 
filter is ResNet’s internal network and the source filter 
is the Internet links) and count the unique destination 
addresses: flow-cat cf05.2000-09-26 * | flow-filter -I1 -i47,62 
| flow-stat -f8] grep -v ’ #’ | we -1. 

This reports 11,739, which is greater than the 
actual number of hosts that generated traffic. This 
implies that the network had been host scanned at least 
once. 


Finally, here is a way to count the Internet hosts 
that have sent traffic to ResNet: 
flow-cat~ef05. 2000-09-26." jf \ 
flew-tilter <=44762 |. 
flow-stat -f9 | \ 
grep -v ’ #’ | we -1 


This reports 879,438, which is less than the num- 
ber of hosts that we sent traffic to (945,189). This 
implies that ResNet hosts were involved in host scan- 
ning the Internet. 

Security Tools 


We most commonly use the flow logs in our var- 
lous security functions, for incident response, 


Fullmer & Romig 


intrusion detection, firewall planning, and security 
assessments and consultation. 


Intrusion Detection 


NetFlow logs are not useful for any form of con- 
tent based intrusion detection since the flow records 
do not contain the data portion of the network traffic. 
We also do not have a detailed picture of the values in 
all of the headers for individual packets, and so we 
cannot use that to detect signatures of intrusions that 
leave calling signs there. We can use the flow logs to 
detect network access policy violations, to report on 
the activities of known suspicious hosts, and to detect 
some of the more obvious forms of scanning and 
denial of service attacks. 


Flow-dscan is an attempt at detecting and report- 
ing interesting network related events in near real- 
time. It can be configured to report: 

e Excessive octets or packets per flow — typically 
floods. 

e A source IP contacting more than a threshold of 
destinations — host scanning. 

e A source IP contacting more than a threshold of 
destination ports on a single host. The port list 
is limited to 0..1023 to keep the memory over- 
head low. 


Flow-dscan must keep a fairly large hash table of 
{source, destination} pairs in memory to be able to 
detect slow port and host scanning. Memory 1s allo- 
cated and managed dynamically based on the fre- 
quency of flow arrival. The ager and various table 
sizes can be user configured. Optional source and 
destination suppress lists are also supported in hash 
format for fast lookups. Typically on-line multi-player 
game servers and web-based add servers must be 
entered in the suppress list to prevent them from being 
reported as scans. Flow-dscan can either be run on 
archived data sets or by connecting to flow-capture for 


>flow-cat /netflow/se2/cf05.2000-09-28.* flow-filter -r6 -il -147,62 \ 


TNt Ob): 


| tf Wowedse ear 7b ops =O) 7B 


port scan: -src=140.2547103762 dst=24 160.184. 73; Starte—966204748 


>flow-cat /netflow/se2/cf05.2000-09-28.* | \ 
flow-filter -r6 -il -147,62 -f flow.acl -S portscan | flow-print 


Sit “SretPaddress Dat DstiPaddress Pr Sree DstPpe CPs Octets 
0001 1404254. 103462 0036 24-1601 84.32 O67ETE. -15 1 60 
OOOl 140-2 54.103 462 003e 24-1 60gm184.32 06 £79 7 al 60 
ODOT? AQ e7 5G LOS WO? OO3e: 24.6041 S432 UGr-ffa. od il 60 
COON, ARO? 5A lO 362 G0 36-224 Vo Oo 4 232 06 7ib. 13 it 60 
OOO01 L407 54% 103 462 OO03e 244160,184.32 OG edb. «L3 1 60 
O00] 1402542 103262 0036 241608184 .32 06 e4c 5d i 60 
0001 140n2542.103 362 O03e 245) 6000 84.32 06 e4d 7 i) 60 
0001 1402542103862 003e 244160m184.32 OGrceaes «5 a 60 


Figure 9: Results of running flow-dscan to detect a scan and flow-print to view the associated traffic. 
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real-time data. Flow-dscan is usually run in the back- 
ground against a live feed of flow records, and reports 
its results through syslog. You can also force flow- 
dscan to run in the foreground and report to stdout, 
which is useful for interactive use against archived log 
files (see the section on intrusion detection). Visually 
inspecting the flows with flow-filter and flow-print 
can verify the scanning activity. 


Normally flow-dscan would be run against a live 
feed of flow records, but you can also run it against 
archived logs. In Figure 9 we use flow- filter to pull 
out TCP traffic from OSU going to the Internet, and 
then run flow-dscan in the foreground on the resulting 
flow records. After running flow-dscan we used flow- 
filter to pull out traffic coming from the source of the 
scan and print It. 


Although it is not practical to respond to every 
trigger it has been useful to log such activities so that 
we can refer back to them at a later date. The fre- 
quency of external sources scanning our network is so 
high that an insecure machine will almost certainly 
end up on someones database of vulnerable hosts 
within 24 hours of installation. Insecure hosts are fre- 
quently compromised within a day of being set up on 
the network. 


A simple script named flow-scan-report takes a 
time range and IP address as arguments and uses flow- 
filter and flow-print to display flow activity to and 
from that address in the given time range. This makes 
it easy to pre-compute brief snapshots of network 
activity for each of the detects that turn up in IDB (our 
Incident Database, in the section “Related Work’’). 


Incident Response 


As we saw with the previous example, flow-filter 
is an effective tool for pulling interesting traffic out of 
the haystack. This is invaluable for incident response. 
For example, if we receive a report that one of our 
computers was involved in a scanning and intrusion 
incident at another Internet site, we can use the flow 
logs to: 

e Confirm whether the alleged incident actually 
involved OSU. 

e If it did, we can usually use the flow logs to 
determine what hosts the OSU host contacted 
by using flow-filter to search for traffic coming 
from the OSU host. 

e We can also search for traffic going to the OSU 
host to determine whether it is being controlled 
from elsewhere. 

e Once we identify the hosts used to compromise 
our hosts, we can search the flow logs for traf- 
fic from those hosts to OSU to discover other 
hosts that might have been compromised. 


Iterating over the flow logs with varying options 
to flow-filter, flow-stat, and flow-print on each pass 
allows us to quickly determine to source(s) and desti- 
nation(s) of DoS attacks and potentially the attacker 
and their arsenal of compromised hosts. Once the 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


The OSU Flow-tools Package and Cisco NetFlow Logs 


compromised hosts, victims, or attackers are isolated 
the IP addresses can be quickly disabled by use of a 
black-hole router which injects specially tagged pre- 
fixes into our backbone routers which get rewritten 
using route-maps to point to a non existant destination. 


For example, early in the afternoon of July 3, 
1999 we were alerted of slow or non-responsive net- 
work services on campus. It didn’t take long to find 
that the inbound side of our OC3 connection to OAR- 
net was full. Running the most recent flow logs 
through flow-filter to isolate inbound traffic and flow- 


stat to create a summarized traffic report by destina- 


tion IP isolated the destination to a single host. A sec- 
ond run of the flow logs through flow-filter to isolate 
flows to that single IP revealed thousands of ICMP 
echo replies — a Smurf attack. Further analysis of the 
destination IP flow logs revealed an IRC client session 
which is a common ingredient on provoking a denial 
of service attack. Disabling the host with a black-hole 
route prompted the attackers to end the attack end 
shortly after. 


Unfortunately the Smurf attack was only a pre- 
cursor to the activity that followed later that day. 
Shortly after one of the evening fireworks displays our 
upstream provider sent a page informing us that severe 
denial of service attacks originating at OSU required 
them to shut down OSU’s Internet connection. Most 
departments at OSU at this time were connected with 
10 megabit per second Ethemet to the campus back- 
bone, so in order to generate a full OC3 of traffic we 
knew that many hosts on many different LAN seg- 
ments must be involved. Again, flow-filter was used 
to isolate outgoing traffic by filtermg on the interface 
fields and flow-stat was used to generate a report 
based on destination IP, and we discovered the IP 
address of a single victim. We soon discovered that 
many hosts on the OSU network were sending high 
bandwidth UDP streams to the victim. Further analy- 
sis of a larger window of the flow logs using flow-fil- 
ter and flow-stat to create a report of source IP 
addresses contacting the victim revealed about 43 
sources on many different LAN segments participating 
in the attack. Using those sources in a filter we also 
found that multiple victims were involved over the 
course of the day. 


Many of the compromised hosts on campus were 
not very active on the Internet, so it was easy to spot 
the IP address of the attacker in the flow logs and the 
TCP port used to start the UDP floods. The attack was 
controlled by a fairly simple set of Perl and shell 
scripts that connected to compromised hosts through a 
shell backdoor on the FTP port, from which it would 
run a script to download a UDP flooder called milk 
from another university through the Remote Copy 
Protocol (RCP). The milk script would then be run 
against one of several external targets. 


Using black-hole routes to disable the compro- 
mised hosts, victims, and attacker proved effective in 
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stopping the attacks. Later long term analysis of the 
flow logs for the compromised hosts revealed the 
method of break-in and several sets of hosts that were 
involved in what was apparently a distributed and 
automated scan and exploit. One last iteration over the 
archived flow records after the incident revealed that 
as many as 250 hosts at OSU were compromised in 
the initial set of breakins on July 2, although only 
about 50 of them were used for the UDP denial of ser- 
vice attack the next day. 


The use of flow logs to home in on compromised 
hosts and their traffic has shown that it is not uncom- 
mon for a site to be scanned for vulnerabilities by one 
host, compromised at a later date by a second, contact 
a third site for downloading of denial of service and 
exploit tools, and then have the installed and waiting 
remote controlled denial of service programs be trig- 
gered at a later date by yet a fourth site to attack a vic- 
tim. 

Firewall Planning and Security Assessments 


As we have instrumented the rest of our core 
routers and switches with NetFlow exports, we have 
come to increasingly rely on the flow logs for guid- 
ance in designing firewalls and in studying the net- 
work behavior of systems in site and product security 
evaluations. 


Host Activity Profiling 


Flow-host-profile builds a list of the network 
services running on each host on campus, and allows 
us to compare activity over a period of time with the 
existing profile to detect changes. We are especial ly 
interested in new hosts and new services that show up 
on campus. The sample output from flow-host-profile 
(Figure 10 shows activity for several services that we 
had not previously seen (e.g., HTTP services on 
128.146.1.4). The new activity on port 7440 on 
128.146.1.3 might be a new service, but sometimes 
busy clients have high end port numbers that show up 
in the report (as they get reused the connection count 
goes up, passing our activity threshold). We’ve been 
experimenting with looking at not just the presence of 
new services (and hosts) but also with changes in the 
level of activity of a service. For example, it would be 
interesting to know if activity on a usually quiet FTP 
server suddenly increases (possibly due to warez trad- 
ing through a writable directory, for example). Unfor- 
tunately, flow-host-profile 1s very sensitive to traffic 
that evades its attempts to winkle out the client/server 
role of each endpoint, and tends to either create many 
false positives or becomes insensitive to low levels of 


IP-ADDRESS PORT SERVICE PROTO CONNECTIONS DAYS 


28 ea Oe ed 53 domain i) 3 
L23e 1468 1.4.2 162 snmptrap 1/7 S 
1288 146¢1.3 7440 N/A 6 3 
128.146.1.4 SOM Tee p 6 3 
128% 14621. 5 JS: nitalk Ly 3 
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Intrusion activity if we turn the thresholds up to 
decrease the false positives. 


Common Problems in Interpreting NetFlow Logs 


We already discussed the difficulty of determin- 
ing the client/server role of the hosts at each end of a 
flow (in the section on general analysis tools) and of 
interpreting the flows in light of the fact that they are 
ordered chronologically by the ending time of each 
flow. There are a number of other issues that you need 
to keep in mind when you are reading through flow 
logs. 


It is important to recognize that source IP 
addresses are easily spoofed. This 1s common in many 
denial of service attacks or in chaff created by scan- 
ning tools like nmap. If you have unicast Reverse-Path 
Forwarding (RPF) checking enabled on your devices, 
or use Access Control Lists (ACLs) to drop spoofed 
traffic this traffic will be recorded in your flow exports 
with a destination interface of 0 (indicating that the 
traffic arrived, but was not forwarded). Of course, 
hosts within a LAN can still spoof the addresses of 
other hosts on the same LAN. 


Note that NetFlow enabled devices only create 
flow entries for the traffic that actually passes through 
that device. If you have asymmetric routing conditions 
where outbound traffic passes through a different 
router than the analogous inbound traffic and you are 
only looking at the flows from one router, you will 
only see part of the flows representing those 
client/server sessions. To get a complete picture of the 
traffic in these cases you would have to combine the 
flow records from the devices that handle all of the 
traffic (through flow-cat and flow-sort). Note that 
since flow-sort uses a 35 minute window for sorting, 
you need to merge the files in chunks that span less 
than 35 minutes. This presumes that the clocks have 
been accurately synchronized to a common time 
source. 


One issue that needs to be considered when 
using flow exports from the routers is the reliability of 
the data sets. Flows are currently exported via UDP 
with no ability for the collector to signal retransmis- 
sion if it detects missing NetFlow PDUs. Each PDU 
contains a 32 bit sequence number that can be used to 
detect missing or out of order flows. Out of order 
flows could be an indication of an attacker trying to 
inject false PDUs into the collector. Missing flows 
could either indicate the flow-collector is overloaded, 
possibly due to an attack or the network segment 


PERCENT 
1 100 
il 3 
1 100 
1 100 
1 0 


Figure 10: Abbreviated output from flow-host-profile, showing new hosts and services. 
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connecting the flow collector to the router is over sub- 
scribed or under attack. The possibility of spoofed 
flows can be minimized by deploying unicast RPF or 
IP spoofing filters appropriately. Overloaded network 
segments can be avoided by directly connecting the 
collectors to a dedicated router interface on the router 
exporting the flows and limiting traffic on that net- 
work with access lists. Cryptographic signatures on 
flow PDUs and a reliable transport mechanism could 
reduce some of the potential problems, but not without 
the memory and CPU penalties on the routers and line 
cards. The potentially unreliable export mechanism 
and 32 bit sequence number is adequate for our cur- 
rent needs. 


Privacy and Legal Concerns 


The flow logs do not contain a record of what is 
usually considered the contents of the packets. This 
means that although we could determine that a given 
host accessed a given web server at a certain time, the 
flow logs would not contain a record of the URL 
requested or the response received. However, if you 
can correlate the activity recorded in the flow logs 
against the data in other logs (such as authentication 
logs), you might be able to match accounts (and so, to 
a large degree, people) to IP addresses, IP addresses to 
their associated network activity, and then match that 
network activity to specific details such as URLs 
requested, email addresses for correspondents, news- 
groups read and so on. Consequently, the act of 
recording and archiving NetFlow records raises a 
number of privacy concems. 


In addition, OSU is a state university and as such 
is subject to the state public records laws, which indi- 
cate that most records created as part of the normal 
business processes of the university are “fair game” 
for disclosure requests from the general public. This of 
course raises a fair degree of concem that someone 
might request a copy of our NetFlow logs to determine 
what our employees are using their computers for. On 
the other hand, we are also subject to FERPA (Family 
Education Rights and Privacy Act) which indicates 
that student academic records are exempt from the 
public records laws. According to our lawyers, it is not 
clear whether the flow records for students would be 
considered protected by FERPA, or whether they are 
part of the public record. Since the flow logs cover a 
mixed population of students and non-students, and 
since we have no easy way to separate them, they 
enjoy a sort of murky, though dubious legal protec- 
tion. 


We try to protect the privacy of our customer 
base by storing the logs on secure servers, with limited 
access by staff members, and with clear access and use 
policies in place. We do not archive the raw logs to 
tape, although we do retain a fairly long window (cur- 
rently about four months worth) on our central file 
server. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


The OSU Flow-tools Package and Cisco NetFlow Logs 


Our rationale is that the logs are invaluable for 
security, performance and network monitoring and 
usage based billing. We could aggregate the data and 
use that for some of these functions, which would 
solve most of the privacy concerns. However, having a 
long (2 to 3 month) window of past logs is invaluable 
for incident response, and we expect that it may prove 
invaluable for bill dispute resolution as well. We think 
that the level of detail present in the flow logs repre- 
sents an acceptable balance between utility and pri- 
vacy for our environment. On the positive side, we 
have found that we have had to do content based sniff- 
ing (e.g., with tcpdump [15]) far less often, since we 
have a ready source of information about network 
activity. 


Related Work 


Other groups have also been working on tools 
for collecting Cisco flow logs. In particular, you might 
be interested in looking at the CAIDA tool collection 
[16]. In particular, the cflowd [12] system provides a 
mechanism to collect, save and aggregate NetFlow 
exports and view a variety of summaries and graphical 
views of the collected data. There are also a number of 
tools that can be used with the cflowd package to cre- 
ate reports and graphs. 


Cisco has also released a set of tools for collect- 
ing NetFlow logs [2]. These tools provide fairly 
sophisticated data aggregation, graphing and billing 
features, and as you might expect, support all of the 
versions of the NetFlow exports, including version 8 
aggregated NetFlows. 


OSU has integrated intrusions detected through 
flow-dscan into its Intrusion Database system (IDB). 
IDB is a web front end to a list of intrusion detects 
from a variety of sources, including host based scan 
detectors, snort and now flow-dscan. Our incident 
response staff reads through the detects in IDB and 
responds to them in a variety of ways. A hook from 
IDB allows the operator to easily view a summary of 
network traffic matching the detect, which is derived 
from the archived flow logs. Incidents that turn into 
full-fledged investigations are tracked through our 
incident tracking system, SITAR [9], which also has 
hooks that allow us to view the results of previous 
searches through the flow logs or to initiate new 
searches. 


David Brumley at Stanford University has writ- 
ten a program that converts argus [1] logs into flow- 
tools format, allowing you to use this as an additional 
source of data for flow-tools [5]. Dave has also written 
a small perl script for detecting scans through thresh- 
old violations [6]. 


Larry Lidz at the University of Chicago modified 
the extract program from the netlog package [4] 
package to read the logs that flow-tools creates [7]. 
The resulting program, flowextract, allows you to 
easily create watch lists to report network activity for 
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known hostile hosts, or for critical hosts in your envi- 
ronment, or to identify traffic that might indicate hos- 
tile activity (for instance, traffic to TCP port 31337). 
Larry has also written a program called flow-merge 
which merge sorts flow logs from multiple routers into 
a single log. This is helpful in cases where multiple 
routing paths lead to asymmetric routing conditions, 
or where you need to search all of your connections to 
the Internet for intrusion activity. 


Simon Leinen has put together a great summary 
of flow related references and tools which is well 
worth looking at [11]. 


Future Plans 


We are often asked why we have not merged our 
work with the cflowd package. There are several rea- 
sons, chief among them being motivation and philoso- 
phy. We have not had a need to use the presentation 
and analysis tools that cflowd provides since we have 
other tools to do that work. Cflowd appears to have 
been designed to facilitate the aggregation of data 
directly from NetFlow exports, and most of the tools 
written to process flow data from cflowd work with 
these aggregations rather than with raw flow data. The 
NetFlow capture mechanism in the flow-tools was 
designed to efficiently store and manipulate flow 
records in compressed form, and data aggregation is 
done by tools operating on these raw logs (if at all). Its 
design also allows us to take a highly distributed 
approach to data collection and analysis. It is possible 
to either write something to export the flow-tools logs 
into whatever format the cflowd package uses inter- 
nally, or to duplicate the incoming UDP NetFlow 
records into parallel incarnations of flow-capture and 
cflowd. 


The principle problems with using the flow logs 
for intrusion detection are that it is difficult to cor- 
rectly determine the client/server role of the two end- 
points of a flow and that the flows do not contain 
packet contents. We have not converted our intrusion 
detection systems to use the sorted flows from flow- 
sort — we expect that this would allow us to both 
increase the sensitivity of flow-host-profile and flow- 
dscan, and also greatly reduce the frequency of false 
positives. 


Two papers by Yin Zhang and Vern Paxson 
describe their work with detecting backdoors [17] and 
detecting stepping stones [18] using packet size, 
packet timing characteristics, and correlations between 
flows of traffic. Aggregating packets into flows 
obscures some of the detailed information that they 
used in their algorithms (individual packet sizes, 
delays between packets), but it might be possible to 
adapt their work to flow based data using average 
packet sizes from the flows and calculated average 
delays, or using flow sizes and delays between the 
flows. We hope to start working on this soon. 


We will soon be adding support for the NetFlow 
version 7 PDU (used on some Cisco switches). Cisco 
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also has a version 8 PDU which 1s used for exports of 
aggregated flow data [3]. We plan to add this at some 
point, but doing so will require development of a dif- 
ferent set of analysis tools, since the current tools are 
designed to work with flows, and not with the aggre- 
gated data in tables that are supplied in the version 8 
PDU. 


We also plan to add support to flow-capture to 
use the sequence numbers in the version 5 and version 
7 NetFlow export headers to detect and report missing 
NetFlow exports. This addresses some of our concerns 
about using the flow logs in incident investigations 
since we will at least be able to tell when critical 
records are possibly missing, or note that there are no 
missing records. 


The current filtering mechanisms in flow-filter 
are sufficient for a wide range of tasks, but it would be 
nice to provide more powerful filters. We plan to fully 
implement Cisco extended access contro] lists. 


We are also working toward building more pow- 
erful interactive front ends for browsing our flow logs. 


Conclusion 


Cisco NetFlow exports and the flow-tools pack- 
age contribute to our ever growing toolbox of network 
management resources at Ohio State University. We 
are collecting and archiving flow exports from twenty 
routers with eleven flow collectors, one file server and 
two high performance data crunching servers. Our 
campus network has over 500 LAN interfaces, 15 
remote sites and multiple external connections includ- 
ing OARnet, Abilene, and a local peering with the 
Columbus cable modem service provider. An online 
archive of 500 Gigabytes of compressed flow exports 
allows turning back the clock on network disruptions 
and security incidents to provide a post-mortem birds 
eye view of events leading up to and surrounding an 
incident. Real-time processing of flow data can shed 
light on difficult to pinpoint activities by allowing us 
to view traffic patterns without having to deploy snif- 
fers or LAN probe devices on every LAN segment or 
WAN link. Departments traffic reports can be gener- 
ated for potential future cost recovery of network ser- 
vices, and over all network usage reports are used to 
determine the impacts of popular new applications 
such as Napster. 


The OSU flow tools are available at 
http:/www.net.ohio-state.edu/software/flow-tools.shtml. We 
also have a mailing list (flow-tools@net.ohio-state. 
edu) — you can subscribe by sending an empty mes- 
sage to flow-tools-subscribe@net.ohio-state.edu. 


Mark Fullmer wrote the bulk of the OSU flow 
tools collection. Steve Romig mostly just uses the 
tools and makes suggestions for further development, 
though he has recently been seen writing documenta- 
tion, fixing various bugs, and finding students to write 
tools like flow-sort, flow-host-profile and flow-con- 
nect. Ron Luman wrote flow-sort and flow-connect. 
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Suresh Ramachandran wrote the flow-host-profile pro- 
gram. 
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FlowScan: A Network Traffic Flow 
Reporting and Visualization Tool 


Dave Plonka — University of Wisconsin-Madison 


ABSTRACT 


Internet traffic flow profiling has become a useful technique in the passive measurement and 
analysis field. The prerequisites for flow-based measurements are now available within the 
network infrastructure — particularly, in popular Cisco network devices. The integration of this 
feature has enabled the “‘flow”’ concept to become a valuable tool for the network administrator, as 
it had been in the past for the researcher. 


This paper describes FlowScan, a software package for open systems that is freely available 
under the terms of the GNU General Public License. FlowScan analyzes and reports on flow data 
exported by Intemet Protocol routers. It is an assemblage of perl scripts and modules and is the 
glue that binds together other freely available components such as a flow collection engine, a high 
performance database, and a visualization tool. Once assembled, the FlowScan system produces 
graph images, suitable for use in web pages. These provide a continuous, near real-time view of 
the network traffic through a network’s border. 


Although there are now a number of tools available that collect and process flow data, there 
is a dearth of visualization tools. By utilizing freely available software tools, FlowScan can be 
readily deployed in most modern educational institution, corporate, and ISP networks. The 
information presented by FlowScan assists in understanding the nature of the traffic that your 
network is carrying. It can be useful in the identification and investigation of anomalies such as 
poor performance and attacks on hosts. It can provide a foundation on which to develop usage- 
based billing or to verify the effectiveness of Quality-of-Service policies. By understanding the 
flows of traffic carried by the network, your institution should be able to make informed network 
management and bandwidth provisioning decisions. 


Introduction 


To better understand the nature of Internet traffic, 
the notion of flow profiling was introduced within the 
networking research community, and subsequently 
extended by other researchers, producing a series of 
useful findings. Flow profiling had been predicted to 
be relevant to applications such as route caching and 
usage-based accounting [ClaffyPB]. Today, based in 
part upon market demands for performance and 
accounting, flow profiling is built into networking 
devices. While not yet standards-based, the flow 
methodology is robust enough to persist through this 
period of vendor-specific implementations, and its 
benefits in many production networks warrant its early 
adoption. 


Network administrators who collect measure- 
ment data often find that they either have collected too 
little data or too much of it. In a sense, flow profiling 
is a “sweet spot’? between those extremes. Flows 
strike a balance between detail and summary. They are 
neither captured packets, nor are they merely aggre- 
gate totals tallied as packets travel across a given port 
or interface. Flows are an expressive abbreviation in 
which each flow represents a series of packets travel- 
ing between “‘interesting”’ end points. While flow fea- 
tures within the network infrastructure are a 
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convenience, the presence of this feature alone is not 
sufficient for reliable continuous use in production 
networks. We need software tools to extract, record, 
and help us understand the flows. 


A variety of tools for flow-based measurement 
have arisen from both the commercial and free soft- 
ware communities [NetFlow], FlowScan 1s one such 
freely available system. FlowScan is an assemblage of 
perl scripts and modules gluing together other freely 
available components, described below. Similar to 
other systems aimed at helping users to make sense of 
an overwhelming quantity of data, FlowScan means to 
simplify the collection, storage, and visualization of 
such data. Like most software tools, FlowScan has 
both ancestors and other relatives that played a part in 
defining its characteristics. 


Several tools have set precedents for collection 
of network traffic data using passive measurement 
techniques. Of the freely available tools, MRTG 
[MRTG, Oetiker] and Cricket [Cricket, Allen] are 
among the most popular. Typically, these tools collect 
measurements by periodically collecting SNMP val- 
ues, such as the interface or port counters named 
ifinOctets and ifOutUcastPkts, from routers and switches. 
They subsequently store this data in a reduced form 
that is easy to manage since it does not burden the user 
with database management responsibilities. These 
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tools are ultimately effective because they make the 
data available to the end user in a useful, convenient 
manner. The design goals of FlowScan are similar to 
those of these tools. 


FlowScan analyzes and reports on NetFlow data 
collected by CAIDA’s cflowd [cflowd], a mature flow 
collection and analysis tool. This flow data is scav- 
enged by FlowScan, which simply tries to discover 
interesting things about those flows, and maintains 
counters that reflect what is found. FlowScan then 
stores this myriad of counter values using Tobi 
Oeitker’s RRDtool [RRDtool]. RRDtool is a a 
database system built from the ground up to effic- 
tively store and report on time-series data. Lastly, 
through the use of RRDtool and other front-ends, 
FlowScan provides reporting capabilities and visual- 
ization of the processed flow data. 


FlowScan’s Functionality 


FlowScan and its component parts are responsi- 
ble for collecting and processing raw flows exported 
from routers. FlowScan examines each flow and main- 
tains counters based upon that flow’s classification. 
FlowScan then periodically reports its results and may 
optionally take other actions. It may be configured to 
either archive or discard the raw flows after process- 
ing. 

FlowScan, in its current form, supplies two 
report modules that illustrate its functionality: Campu- 
sIO and SubNetIO. CampusIO 1s a full-featured report 
module that is often the first and only report run by 
most FlowScan users. As such, its features are often 
thought to be those of FlowScan itself. The CampusIO 
report interrogates the raw flows, accumulates total 
counts and pushes these numeric statistics into high 
performance time-series Round Robin Databases 
[RRDtool]. Each database contains packet, byte, and 
flow counters. These counters are maintained in both 
in and out directions when appropriate. 


Each Round Robin Database created by FlowS- 
can contains between one and eight traffic statistics, 
stored at five-minute intervals, based upon one of 
these flow attributes: 

e the IP protocol such as ICMP, TCP, and UDP 
e the well-known service or application such as 
ftp-data, ftp, smtp, nntp, http, RealMedia, 

Quake, and Napster 

* the class A, B, C network, or CIDR block in 
which a “‘local”’ IP address resides 

e the AS (Autonomous System) pair between 
which the represented traffic was exchanged 


Additionally, FlowScan maintains general traffic 
databases that contain total, multicast, and traffic 
involving unknown networks. 


Figure 1 1s a sample FlowScan graph of our cam- 
pus traffic over a period of 24 hours, demonstrating 
some of the information handled by CampusIO. 
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This graph shows several features: 

e There is a circadian rhythm to our campus traf- 
fic, with the low point centered about 6 AM 
and with peaks in the late evening. 

e There is more total outbound traffic than 
inbound. That is, our campus consistently pro- 
vides more Internet content than it consumes, 
regardless of whether or not our web cache 1s 
enabled. 

e There 1s a significant amount of outbound FTP 
DATA content. This metric is a combination of 
both ftp-data and PASV mode ftp data transfers. 

e Napster users were responsible for an amount 
of traffic both inbound and outbound that rivals 
or exceeds both general web traffic (HTTP) and 
file transfer traffic (FTP) [Plonka]. 

e Some sort of Napster outage occurred at 
approximately 3 PM local time. 
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Figure 1: Graph of U W-Madison’s campus traffic I/O 
during 24 hours, Sep 18 & 19, 2000. 


The SubNetIO report requires a bit more config- 
uration on the part of the installer, but 1t subsequently 
provides all the functionality of CampusIO, plus it 
maintains per-subnet statistics for applications such as 
billing a given campus “customer” based on their 
subnets’ usage of precious bandwidth in and out of the 
campus Itself. 


Background on Flows and Cisco NetFlow 


FlowScan utilizes flows defined and exported by 
Cisco’s NetFlow feature. This flow definition 1s essen- 
tially that which was introduced in [ClaffyPB]. By this 
definition, an IP flow is a unidirectional series of IP 
packets of a given protocol, traveling between a 
source and destination, within a certain period of time. 
The source and destination are defined by IP address. 
Because the flow is unidirectional, nearly all useful 
exchanges between two hosts, such as a client and a 
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server, are represented by at least two flows — one 
flow in each direction. For TCP and UDP flows, this 
definition considers the port number to be part of the 
source and destination address, making it convenient 
to determine which “well known” application such as 
HTTP and FTP is likely to have been responsible for 
the traffic represented by a flow. Additionally, Cisco 
NetFlow PDUs include the router’s source and desti- 
nation interface (named by the SNMP “‘ifIndex”’) and 
NetFlow version 5 PDUs include source and destina- 
tion netmask and Autonomous System which are of 
interest to LAN and WAN engineers, respectively. 
FlowScan-1.003 requires Cisco NetFlow version 5 
PDUs and makes use of all of these flow attributes. 
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Figure 2: An image of Cisco’s NetFlow V5 PDU 
header [McRobb]. 
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Figure 3: An image of Cisco’s NetFlow V5 PDU 
entry [McRobb]. 


Note that while the term “flow” refers to the 
series of packets itself, NetFlow and FlowScan users 
often refer to the IP traffic flow accounting record that 
is exported from the router, and subsequently ana- 
lyzed, as a “‘flow’’. This usage is natural because it 1s 
this exported accounting data that is the tangible data 
object that FlowScan manipulates. So, in this paper 
the term “flow” will most often have the latter mean- 
ing. Remember that this usage differs from the formal 
“flow” definition used in some other written works 
such as [ClaffyPB]. 


Cisco’s NetFlow version 5 flow export format 
[Cisco]: 
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Version 5 flow-export packets contain a 
flow header followed by a number of flow 
entries. The number of flow entries in the 
packet is in the count field in the flow 
header. 


Unlike version 1 flow-export, version 5 
flow-export has AS numbers and netmask 
lengths for the source and destination. 
[McRobb] 


While the NetFlow V5 PDU is well documented 
in [McRobb] and [Cisco], Figures 2 and 3 are here as 
a catalog of the flow attributes that are available to 
FlowScan. This set of attributes enables FlowScan’s 
current capabilities, and to a degree ultimately limits 
them by imposing performance requirements and mea- 
surement compromises as described in the section on 
FlowScan problems. 


FlowScan’s Architecture 


Hardware 


A diagram of a basic FlowScan system’s hard- 
ware components is show in Figure 4. 


isco 
TelcoLAN Rautor 





FlowScan graphs 
< via HTTP, etc. 


Heer 





<=v—P cflowd/FlowScan host 
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Figure 4: An image showing the basic hardware for 
the FlowScan System. 


While FlowScan does not have strict platform 
requirements, most users who have successfully 
deployed it have dedicated either a SPARC machine 
running Solaris or Intel machine running GNU/Linux 
or BSD as a FlowScan system. Deploying on one of 
these platforms is also convenient because cflowd 
builds and runs on them as well, which allows one to 
co-locate FlowScan with cflowd so that both have 
access to the local disk. The fastest FlowScan 
machines appear to all be multi-processor Intel 
machines. 


For further information, the installation docu- 
mentation provided in the FlowScan distribution pack- 
age discusses some recommended hardware parame- 
ters such as disk space and network interface card. 


Software 


A FlowScan system consists of a number of so ft- 
ware components. The first such component is cflowd 
[cflowd], which is_ described thoroughly in 
[McRobb2]. FlowScan uses cflowd strictly as a flow 
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collector. As such, the cflowd components used by 
FlowScan are the cflowdmux and cflowd programs. 
cflowdmux receives UDP Cisco version 5 flow PDUs 
from routers and and passes them to cflowd which 
writes them to disk in a portable, well-defined format 
of its own. FlowScan requires that the installer apply a 
patch to the cflowd sources. This modification enables 
cflowd to rotate and time-stamp its flow files at 
FlowScan’s pre-defined sampling interval, which is 
typically five minutes. The decision to use five-minute 
samples was influenced by the popular tool MRTG 
[MRTG]. 


The second component is a program called flows- 
can; note that its name consists of only lower-case 
characters. ““FlowScan” 1s the package whose primary 
procedural component is the program flowscan. This 
program is a perl script that is the central process in 
the system. It loads and executes report modules of the 
administrator’s choosing. These report modules are 
simply perl modules that are derived from the FlowS- 
can class defined in the FlowScan.pm perl module. 
FlowScan reporting modules are described below in 
the architecture section. As such, it is the flowscan 
script that actuates the whole system by maintaining 
databases of statistics regarding the IP traffic repre- 
sented by the flows. 


The third major component of FlowScan 1s 
RRDtool. RRDtool is described in [RRDtool] and is 
well documented in the supplied on-line manual 
pages. The FlowScan system uses RRDtool to store 
numerical time-series data and automatically distill or 
aggregate it into averages over time. Using RRDtool 
in this manner essentially replaces cflowd’s artst++ data 
aggregation features, which have a different API, and 
no integrated graphing features such as those built into 
RRDtool. Specifically, RRDtool is used by FlowS- 
can’s supplied report modules to maintain a set of 
RRD files that form an extensive database of IP flow 
metrics. Also, RRDtool and RRGrapher are responsi- 
ble for producing output such as graphs of IP traffic as 
GIF or PNG format image files. 


The other components of the FlowScan package 
are utilities such as the make command, the Unix cron 
job-scheduling facility, and the gzip compression util- 
ity. Also, there is the flowdumper utility, supplied with 
the Cflow package, which is used to examine the raw 
flows “‘manually”’. 


Figure 5 is a diagram of FlowScan’s components 
and the data objects on which they operate. 


FlowScan uses the disk as a large buffer area in 
which cflowd writes raw flow files that wait to be post- 
processed by flowscan. This buffering is an important 
fail-safe when used in networks with very high traffic 
or flood-based DoS attacks because FlowScan some- 
times develops a backlog of flow files yet to be pro- 
cessed, which could total gigabytes in size. 


Cisco’s NetFlow was chosen as the base technol- 
ogy because of the ease of development within our 
campus’ existing infrastructure. 
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Together with cflowd patches [patch], FlowScan 
enables multiple flow log analyses in one pass: 


« flow archiving for post mortems 

e real-time analysis with cflowd 

e near real-time or post-processing with FlowS- 
can 


FlowScan System Overview 


Figure 5: An overview of the software components of 
the FlowScan System. 


Anatomy of a FlowScan Report Module 


A FlowScan report module is an object-oriented 
perl module that has FlowScan as its base class. The 
module named FlowScan.pm implements the base 
reporting class, and serves as an example of what a 
FlowScan report must provide. Objects of a FlowS- 
can-derived class have 3 methods, a.k.a. subroutines. 
These are: 
perfile — Before flowscan processes a given time- 

stamped raw flow file produced by the patched 

cflowd, it invokes the perfile subroutine. As such, 
it is called once per file. The name of the file 
about to be processed is passed as an argument 
to this routine. For the convenience of derived 
classes, perfile also converts the time-stamp 

embedded within the raw flow file’s name to a 

native Unix time_t representation. The names of 

the raw flow files produced by the patched 
cflowd are of the ‘flows. YYYYM- 

MDD_HHMISS+TZ” format. For example, 

‘“flows.20000917_20:08:14-0500”’ is a file created 

at 8:08 PM on September 17, 2000 in a locale 

that is five hours west of GMT. 

wanted — As flowscan reads each of the raw flows 
from within a cflowd-produced flow file, it 
invokes the wanted routine once per flow. This 
routine decides whether the current flow 1s 
wanted, 1.e., whether it is an interesting flow for 
the report. The wanted subroutine is at the heart 
of a FlowScan report because it provides the 
opportunity for a report to interrogate the val- 
ues stored in each flow and act accordingly. 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


Plonka 


report — After flowscan processes the last raw flow in 
a given flow file, it invokes the report routine. 
This routine 1s responsible for dispatching with 
any information “‘discovered”’ and collected by 
the aforementioned wanted routine. As such, 
this routine reports what has been discovered 
about the flows analyzed within the current 
flow file. 


Two report classes are supplied with FlowS- 
can-1.003: “CampusIO” and “SubNetIO”. Campu- 
sIO is implemented in the file CampuslO.pm, following 
the usual naming convention for perl modules that 
implement object-oriented classes. CampusIO is 
derived from the FlowScan class defined in FlowS- 
can.pm. SubNetIO is, likewise, implemented in SubNe- 
tlO.pm and its base-class is CampuslIO. That is, SubNe- 
tIO is an extension that relies upon the functionality 
provided by CampusIO. As such, it is appropriate to 
run CampusIO or SubNetIO, but not both, since Sub- 
NetIO’s functionality is a superset of the CampusIO’s 
functionality. 


Figure 6 is a simplified representation of the 
logic of CampusIO in perlish pseudo-code. 


One of the features unique to the FlowScan sys- 
tem is its modular reporting structure. Because most of 
what the system does is the responsibility of periph- 
eral modules, the flowscan script essentially just 


package CampusI0O; 


sub perfile { 
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provides a framework for arbitrary testing and peri- 
odic reporting on flow content. 


Stateful Inspection of Flows 


The CampusIO report uses a number of heuris- 
tics that help it to identify elusive traffic, such as that 
of the Napster application [Plonka] or of PASV mode 
ftp file transfers. These heuristics employ a method of 
stateful inspection, which is essentially a variation of 
the stateful inspection of packets employed by many 
modern firewalls. Such firewalls track the state of an 
application session by observing information within a 
packet or series of packets. This technique enables the 
firewall to filter packets according to whether or not a 
session has been established and is still active. This 
state information is gleaned via passive inspection of 
either the packet header or application payload. 


Similarly, FlowScan attempts to track the state of 
an application session, or series of sessions, by 
observing the information within flows. This flow- 
based stateful inspection enables counters to be main- 
tained and reported upon for flows which would other- 
wise be left unidentified or be misidentified. For 
example, the identifying of traffic using a simple test 
of protocol and port number (i.e., tcp and port 6699) to 
identify Napster data flows is likely to sometimes 
erroneously match flows from applications such as 
PASV mode ftp which negotiate dynamic, unprivileged 


## Remember the time-stamp from the filename. 


whence = filename2time_t (filename) 


} 


sub wanted { 
if (exporter_hop(flow::next_hop)) ‘4 


# This flow is destined for another "local" flow-exporting router. 
fF Wel) -catch-it Tater, if and when it*s exported by that rotiter, 


return 0 
} 


4f (outbound_hop(flow::next_hop) or outbound _interface(floweptoutput_if)) { 


# This flow is outgoing. 
outbound_totaltt 


elsif (inbound _address(flow::destination_address)) { 


# This flow is incoming. 
inbound_totaltt 
}' else { 


# This flow (an "intranet" flow) is unwanted. 


return “0 


Te td oy al 


} 


sub report { 


update_RRD_files(whence, inbound_total, outbound_total) 


} 


Figure 6: The logic of CampuslO. 
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port numbers. We minimize or eliminate this error by 
employing a more complicated test based on stateful 
Inspection. 


Figure 7 1s a pseudo-code example of the logic 
used to detect Napster flows and PASV mode ftp data 
flows, which demonstrates the technique. 


Also, without this sort of analysis, a large per- 
centage of our campus traffic would be simply labeled 
as ““unknown’’. As it is now, still more than 30% of 
our campus traffic remains unclassified. 


FlowScan is not the only package that performs 
such stateful inspection of flows. Although FlowScan 
has for some time performed such inspection to iden- 
tify Real Media flows and Napster flows, such a tech- 
nique was developed independently by Simon Leinen 
for Fluxoscope [Fluxoscope] to identify PASV mode 


sub Napster_wanted [ 
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ftp data flows, a feature only more recently added to 
FlowScan. 


Life with FlowScan 


FlowScan produces a variety of graphs that pro- 
vide a different view of network traffic than that pro- 
vided by most other tools. For example, flow-per-sec- 
ond graphs are often as useful in network management 
as are packet-per-second and byte-per-second (band- 
width) graphs. FlowScan’s RRD files are configured 
in such a way that they aggregate counters into less 
granular totals over time. For instance, five minute 
samples are combined into 30 minute samples, then 
into two hour samples, then into 24 hours samples. As 
a result, the sort of anomalies which we are able to 
discover is determined by the time length of the graph. 
In general we have found that by graphing over both 


if (ICMP != flow::protocol || not (TCP = flow::protocol and 
(O24 “<flow:ssreport and L024 <tlowsrdstport)) 4 


return. 0 


} 


## flow is either ICMP or TCP on unprivileged ports 


af ‘(inbound (flow) 
darectron.=  *ans 
OUtsidGeaddr = "flow ssreaddr: 
insidevaddr = flow: :dstaddr 
lielsif Goutbound (flow): { 


direction, =“ outs 
Outside Addr s— low: sdstaddr-. 
inside: addr = Liow: :sreaddr 


if ATCE == flows <prococol-and ACK -& tlow:istep tlags and 


iss. napserver (outside addr)) -{ 


## flow involves an ouside host that is a "publicly advertised" napserver 
remember_napster_server(outside_addr, flow::endtime) ; 
remember_napster_user(inside_addr, flow::endtime) 


beelsit Cis napuser (inside addr) )- 4 


4 flow involves an inside host that has talked to a napserver recently 


it  CCNCE == “flows protocol and 


Napster ports (ft lowrssrcport.,.. 1 low: sdstport)) tor 

(TCMP-==“tlows protocol and. 238 == flow: bytes/flow:: pkts))) 
## Confidence is high that this is a Napster application flow because 
+ tlow t6 eather TCP-on Napster “default” ports or 
## flow is ICMP using the "known" Napster ICMP packet size. 


napster_totaltt; 
return 1 
|} else { 


# Confidence is lower that this is really a Napster application flow 
## because the port numbers are not Napster defaults or the ICMP 
## packet size isn’t right. We’ll keep a count of these anyway. 


maybe_napster_totalt+t 


} 


return 0 


Figure 7a: Pseudo-code logic to detect Napster flows and PASV mode ftp data flows, part 1. 
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short and long terms to be useful. However, each 
serves its own purposes. 


Short-Term Analysis 


By default, FlowScan’s window for short-term 
analysis is 48 hours, that allows easy comparison of 
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supplied with FlowScan provide views of traffic by 
network or subnet, by application or service, and by 
Autonomous System over the past 2 days. In each of 
these three graph categories, graphs are available by 
bits-per-second, packets-per-second, and _ flows-per- 


those two consecutive days. The standard graphs second. 
sub ftp_PASV_wanted { 
i Ur Cr = “t lows protecow): x 
return OQ 
} 
## flow is TCP 
if (not (2) -——-t low <sreport.or 21 == flow: tdsteport on 
(hO024-<=tlowrtsreport and 1024<=" flow dstport) 9" y 


return. .0 
} 
+ flow as either ftp (eontrol. port 21). or 
# TCP on unprivileged ports (>= 1024) 
it “ClO24 <= flows report. and 1024 <= flows "astport) 4 

# could be ftp PASV data flow 

if. “rtp sessronuexicts (tlowessneaddr, 

flow: :dstaddr, 
flow::endtime)) { 
# flow is probably ftp PASV data 
frp PASVvotal ter 

} 

return l 
} 
+ flow 2 -fep. (control, port. 21) 
if (ACK & ‘flow s:tcp. flags:) 4 

+ €tp (control port 21) “stream is active 

remember_ftp_session(flow::srceaddr, flow::dstaddr, flow::endtime) 
ot CRIN Gat tows) tepet bags): 

# ftp (control port 21) stream is closed 

remember_ftp_session_closed(flow::srcaddr, flow::dstaddr, flow::endtime) 
} 
# this flow wasn’t ftp PASV data 
return 0 

} 
sub perfile |[ 

ieatware 
# Forget Napster server hosts (in the outside world) and user hosts 
# (on the inside) that haven’t talked Napster recently. 
forget_quiet_napservers(30*60) ; 
forget_quiet_napusers(30*60) ; 
# Forget ftp session if we have seen FIN and 15 mins has passed 
## since we don’t expect DATA flows so long after control port 21 
# has closed. 
# Also, forget ftp session if we haven’t seen ACK on its control 
# port 21 in the past 30 mins since it’s likely to have timed 
# out by now. Perhaps we missed the flow which indicated that 
## the control port 21 stream has closed. 


forget_closed_or_quiet_ftp_sessions(15*60, 30*60); 


i 


Figure 7b: Pseudo-code logic to detect Napster flows and PASV mode ftp data flows, part 2. 
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The graphs over a short, recent time frame are 
based upon the data that FlowScan keeps at five- 
minute intervals. The coarser-grained averaging in the 
longer term graphs will often hide anomalies seen in 
these finer-grained graphs. Network abuse, such as 
flood-based Denial of Service attacks, are easily visi- 
ble. 


Specifically, in our experience with FlowScan, 
we have learned that a discrepancy between the num- 
ber of inbound and outbound flows or packets is an 
indication of abusive traffic, such as a DoS flood. Sud- 
den changes in packet counts, especially when con- 
strained to one protocol, are usually indications of a 
flood as well. Figure 8 1s an example graph based on 
flow counts representing a flood of traffic which was 
unnoticeable on bandwidth usage graphs and nearly so 
on packet count graphs. The traffic responsible for the 
spikes in TCP flows was a flood of incoming 40-byte 
TCP ACK packets to which the campus host to which 
they were directed responded with 40-byte TCP RST 
packets. Even though a flood of small packets with 
dynamic source addresses produces spikes of equal 
magnitude in both packet and flow graphs, it is more 
readily noticed in the flow graph because the spike is a 
much larger proportion of the total flows than it is of 
the total packets. 
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Figure 8: Graph produced 2000/09/24 showing a 
40-byte TCP DoS flood. 


When visualizing over the short term, it is impor- 
tant to remember that FlowScan increments traffic 
counters at the time when the flows are exported. 
More precisely, it records the values of these counters 
with the time-stamp corresponding to the five-minute 
interval in which cflowd wrote the flow to the raw 
flow file. As such, the time reported in the graphs is 
some function of the flow end time, but may involve 
timeouts defined in the NetFlow implementation. This 
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time-stamp 1s thus not necessarily that when the repre- 
sented traffic was actually observed by the router. 


It 1s possible for a FlowScan graph to report a 
quantity of traffic in a given five-minute period that 
exceeds that actually forwarded by the router. On 
occasion, the quantity reported could even exceed the 
physical capacity of the link that carried 1t, which can 
be misleading. The reason that FlowScan operates in 
this way is twofold. First, for the sake of simplicity, 
FlowScan assumes that the traffic represented there-in 
must have occurred within the given five minute 
period. Without this assumption, we couldn’t simply 
accumulate totals, and plot those values vs. time. Sec- 
ond, while NetFlow flows contain start and end time 
information for the packets in a given flow, they do 
not contain any hint as to how the delivery of those 
packets was distributed between the flow start and end 
time. So, even if FlowScan attempted to record the 
real time at which traffic was observed, it would not 
necessarily be any more accurate. We will see that the 
effect of this “time kludge” is negligible as data is 
coalesced into less granular time samples and visual- 
ized in longer-term analyses. 


Long-Term Analysis 


When producing FlowScan graphs, one may sim- 
ply specify the number of hours over which to plot, 
such as 24 (hours) times 365 (days). Increasing the 
hours significantly beyond the default becomes an 
exercise In customization, since these graphs often 
need to be annotated with dates and other descriptions 
so that they have meaning to their intended audience. 


Daily averages, which are used when graphing 
anything but fairly recent time ranges, hide spurious 
abuse activity such as flood-based Denial of Service 
attacks because such attacks are usually short-lived 
and will be severely minimized by averaging. As such 
the primary value of graphing over extended periods 
with daily averages is to aid in capacity planning and 
perhaps to help target traffic shaping efforts. 


Campus 1/0 by feteork, bits per second 


Vehound(-), eutbound(+) bits/sec 
1 
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Figure 9: Graph produced 2000/09/21 showing cam- 
pus traffic by network over the past 550 days. 


Figure 9 shows a sample FlowScan long-term 
graph using daily averages over 550 days. 


Information gleaned from this graph includes: 
e The academic calendar dramatically influences 
the traffic levels, but only with respect to traffic 
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to and from ResNet, the residence halls net- 
work. 
e There has been an increase in outbound ftp traf- 
fic from the Computer Sciences department 
within the past year. 
e While our outbound traffic level has consis- 
tently exceeded our inbound traffic level, the 
discrepancy between the two appears to be 
increasing. 
Custom Graphs 

In addition to using the “canned” graphs sup- 
plied with the FlowScan distribution, FlowScan users 
can produce custom graphs with the companion tool 
named RRGrapher [RRGrapher]. RRGrapher 1s the 
“Round Robin Graph Construction Set” and is 


rrdtool graph /your/file/name/here.gif \ 


eStart. “Aug 15, 2000 \ 
=Sand. “sep s 15,2000 \ 
--vertical-label ’bits/sec’ \ 
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implemented as a single perl CGI script, which runs 
under a web server that has access to RRD files. 
RRGrapher is also general front-end for RRDtool that 
allows users to interactively build graphs of their own 
design from any RRD files. Since it is based on RRD- 
tool, not FlowScan, it allows one to generate graphs 
containing data from RRD files containing data from 
any RRDtool-based systems such as MRTG, Cricket, 
and FlowScan. 


Figure 11 1s a sample graph produced using 
RRGrapher. This graph shows the bandwidth used by 
ftp data transfers from ftp servers to ftp clients both in 
and out of the campus as determined by FlowScan. 
Figure 10 is the resulting RRDtool command that 
RRGrapher wrote and executed intemally to produce 


=“title *ftp PASV and ftp-data, toutbound/-inbound’ ~\ 
‘DEF; A=f£tp-datasre.rrdsin. bytes: AVERAGE’ -\ 

"DEF: B=ftp-data_src.rrd:out_bytes:AVERAGE’ \ 
’DEF:C=ftpPASV_sre.rrd:in_bytes:AVERAGE’ \ 
'DEF:D=ftpPASV_src.rrd:out_bytes:AVERAGE’ \ 


“CDE E=Av oe." cule A 
'CDERYF=CaO.¢ ae \ 
*CDEF:G=B,8,*’ \ 
*CDEF:H=D,8,*’ \ 


"COMMENT:A) ftp-data_src AVERAGE in_bytes’ \ 


"COMMENT: \n’ \ 


"COMMENT:B) ftp-data_src AVERAGE out_bytes’ \ 


"COMMENT: \n’ \ 


*COMMENT:C) f£ftpPASV_src AVERAGE in_bytes’ \ 


’COMMENT: \n’ \ 


"COMMENT:D) ftpPASV_src AVERAGE out_bytes’ \ 


"COMMENT: \n’ \ 


‘AREA: E#OOOOFF:ftp-data sre in (A,8,*,-1,*)’ \ 


"GPRINT:E:MIN: (min=%.11£%S’ \ 
’GPRINT: E: AVERAGE: ave=%.11£%S’ 
’>GPRINT: E:MAX:max=%.11£%S)’ \ 
"COMMENT: \n’ \ 


“STACK* PPOOR PEE 1 ip-PASW, sree in: (G.7e,0" = 1a) 


"GPRINT:F:MIN: (min=%.11£%S’ \ 
°GPRINT: F: AVERAGE: ave=%.11£%S’ 
*GPRINT: F:MAX:max=%.11£%S)’ \ 
"COMMENT: \n’ \ 


*AREA:GH#OOFFOO:ftp-data src out (B,8,*)’ \ 


*GPRINT:G:MIN: (min=%.11£%S° \ 
"GPRINT:G: AVERAGE: ave=%.11£%S’ 
*GPRINT:G:MAX:max=%.11£%S)" -\ 
"COMMENT: \n’ \ 


’ STACK: H#A0522D:ftp PASV src out (D,8,*)’ \ 


’GPRINT:H: MIN: (min=%.11£%S’ \ 
"GPRINT:H: AVERAGE: ave=%.11£%S’ 
’GPRINT:H:MAX:max=%.11£%S)’ \ 
COMMENT: \n’ 


Figure 10: RRDtool command created by RRGrapher. 
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the graph. At the user’s option, RRGrapher displays 
this command when it produces the graph, allowing 
the user to cut and paste it elsewhere, perhaps to 
schedule it as a batch or cron job. 


ftp PASV and ftp-data, +autbound/- inbound 


bits/sec 





: =f “a f 
Week 33 Week 34 vrek 3G Week 36 


A) ftp-data_src AVERAGE iIn_byces 

B) ftp-data_src AVERAGE out_bytes 

C) FtPPASVY_src AVERAGE in_bytes 

©) ftPPASY_srC AVERAGE out_bytes 

BM frtp—date src in (A,6,",-1,°) (min=-3.24% avee-2.0— oax=0, 1 
@ ftp PASY src in (¢,6,7,71,") (win=-16.9N ave=2.0M max=-0.41 
Mi ftp—date src out (8,6,") (mine1.9N ave=9,6M pax=23, 6M) 


@ ftp PASY src out (0,6,") (ain=0.9% ave=4,64 max=i0, 6M) 





Figure 11: Graph showing campus ftp server traffic 
Aug 15, 2000 through Sep 15, 2000. 


FlowScan Problems 


FlowScan, while still extremely useful in most 
production networks, has exhibited a number of lim1- 
tations and illuminated a number of problems. These 
may be instructive in the building of next generation 
flow analysis tools. 


FlowScan’s near real-time processing lags 
behind when processing flow files containing mostly 
“pathological” flows, such as those flows which rep- 
resent only one packet per flow. These degenerate 
flows are produced to represent the flood of traffic 
produced during most Denial of Service attacks 
because the source or destination information 1s forged 
so that each packet contains a different IP address or 
port number than that which preceded it. In these situ- 
ations the flow export rate approaches the router’s 
packet forwarding rate, resulting in an avalanche of 
flows directed to the collection host. Figure 8 is an 
example graph of such traffic. Unless such traffic 
eventually ceases within hours or 1s blackholed by the 
network administrators, FlowScan would fall hope- 
lessly behind, since it cannot process flows at rates 
approaching the packet forwarding rate of high perfor- 
mance routers with high capacity links. 


Furthermore, under “normal” circumstances on 
some networks, FlowScan can still become buried in 
data. Even with the orders-of-magnitude reduction in 
the number of flows vs. the numbers of packets that 
those flows represent, flow processing tools may be 
Overrun with too many flows to process in close to 
real-time on fast links. Currently, with Cisco’s version 
5 flow-export, FlowScan might not be able to scale 
beyond monitoring a couple fully-utilized OC3 (155 
Mb/s) links. As technologies such as Gb ethermet and 
optical switching are deployed in border routers, the 
packet forwarding rate at these aggregation routers 
could outpace the performance gains in commodity 
personal computer hardware that are important for 
flow post-processing. Specifically, the performance 
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adequacy of the processor and bus to persistent stor- 
age 1s in question. So, as packet forwarding perfor- 
mance improves, FlowScan-like processing may need 
to move from the network’s DMZ to a location much 
closer to the end-user. This migration will require the 
deployment of arrays of measurement equipment. 
Managing an array of such devices will present new 
challenges for the network administrator. 


In part, for tools other than FlowScan, these 
backlog issues have been avoided by aggregating 
totals on the router itself and then exporting those 
totals less frequently as summary PDUs. Another 
alternative 1s to adopt SNMP polling-based gathering 
of flow statistics rather than collecting unsolicited, 
exported flow data. However, both of these methods 
prevent nearly all the interesting post-processing that 
FlowScan currently performs, and also eliminate the 
archivable record of network traffic that detailed flow 
PDUs provide, which has proven invaluable during 
investigations of security compromises and network 
abuse. 


Other problems occur when either NetFlow 
export or FlowScan is misconfigured. In complicated 
environments, configuring Cisco routers for cflowd 
has been easy to do incorrectly, especially since ip 
route cache flow should be enabled only on the appro- 
priate interfaces. Misconfiguration can result in miss- 
ing, skipped, or duplicated flows. Obviously, such 
misconfiguration may result in inaccurate flow statis- 
tics which, unfortunately, are dutifully stored into 
RRD files by FlowScan. These files are FlowScan’s 
sole means of keeping archive data to answer histor1- 
cal questions or examine long-term trends. To combat 
this problem, in our use of FlowScan, we maintain a 
diary of interesting events for correlation with spuri- 
ous graph anomalies. 


Finally, identifying flows as representing traffic 
for specific applications is becoming increasingly dif- 
ficult. Traditionally, the packets and flows of “‘well- 
known” applications were easily identified by interro- 
gating values in the IP header such as port number. 
However, this identification 1s complicated by the 
increasingly popular technique of dynamically negoti- 
ating ports for connections, as well as the use of unre- 
served port numbers by many modern applications. 
Since flows do not contain the packet payload, we are 
not always able to label a given flow by application 
name with a high level of confidence, but instead must 
resort to compromises and heuristics. Furthermore, as 
VPNs, tunneling, and encryption become more com- 
mon, network applications will intentionally hide the 
payload and even the IP header. Quality-of-Service 
initiatives may address some of these passive mea- 
surement problems since the same criteria by which 
packets will be marked for QoS purposes may be suf- 
ficient for flow identification as well. The identifica- 
tion of a flow’s user and service class may have to suf- 
Lice: 
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Future Directions and Possibilities 


Alerts 


Automatic event notification or alert capability 
would be a useful addition to FlowScan. A report 
module named “‘CampusIOAlert”’ is currently under 
development which dispatches alerts to email 
addresses or alphanumeric pagers. We have focused 
on alerts based on tests that would be unlikely to gen- 
erate false positives and would not need to be exe- 
cuted more than once per flow file. For instance, we 
have implemented an alert to report flood-based 
Denial of Service attacks based simply on the detec- 
tion of an imbalance of inbound versus outbound 
flows. 


The opportunity for alert modules to perform 
tests on individual flows may also be useful, although 
more processing intensive since such tests must be 
executed once per flow rather than once per flow file. 
For example, we have had good luck in identifying 
hosts that were infected with a specific email-attach- 
ment worm because this worm attempted to negotiate 
an HTTP connection from the infected host to a well- 
known destination host. 


Ultimately, with a report such as Campu- 
slOAlert, FlowScan users will be able to write their 
own alert tests using the expressive syntax of the perl 
language itself. These alerts will also be able to test 
values from RRDtool’s rrdtool graph command, which 
has expressive RPN expression evaluation and test 
features of its own. 


Performance Enhancements 


One of the largest challenges has been dealing 
with flow file backlog created whenever the quantity 
of flows exported by the router, and subsequently 
written to disk, exceeds the quantity that FlowScan is 
able to process in near real-time. 


Converting FlowScan into a multi-threaded, or 
simply multi-process, application was thought to be a 
viable solution to the backlog issue, and was recently 
implemented by Alexander Kunz, an ambitious 
FlowScan user. This multi-process flowscan enlists the 
help of another flowscan process whenever FlowScan 
had more than one flow file yet to be processed. A 
mutex technique in which each process takes a num- 
bered “ticket” was employed to ensure that subse- 
quent updates of the RRD files occurred in the proper 
time-series order. 


At the time of this writing, September 2000, the 
challenge of integrating this multi-process patch into 
the mainline FlowScan distribution has been delayed 
in favor of simply improving the performance of the 
single-threaded model. A faster single-threaded solu- 
tion avoids those complications inherent in multi- 
threading which we have yet to address, namely that 
the FlowScan code would need to enforce serialized 
access to its internal data structures used for stateful 
inspection. Also, light-weight thread support in perl, 
while available, is still considered experimental. 
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Instead, to vastly improve FlowScan’s single- 
process performance, IP address prefix matching was 
reimplemented with Patricia Trie lookups in a C lan- 
guage extension to perl — 1.e., a perl module. The term 
“Trie’’ 1s derived from the word “retrieval”? but is 
pronounced like “‘try’’. Patricia stands for ‘Practical 
Algorithm to Retrieve Information Coded as Alphanu- 
meric” and is thoroughly described in [WrightS]. The 
Patricia Trie performance characteristics are well- 
known as it has been employed for routing table 
lookups within the BSD kemel since the 4.3 Reno 
release [Sklower]. We chose the Patricia data structure 
and search algorithm upon realizing that FlowScan’s 
decisions based upon IP addresses are nearly identical 
to those decisions that an IP router must make when 
delivering a packet based upon destination IP address. 


Multi-vendor Compatibility 


While nearly all the focus so far has been on 
Cisco’s NetFlow feature, other network hardware ven- 
dors have developed similar flow-based accounting 
technology. Juniper, with its announcement of its 
JUNOS software release 4.1R1 in August 2000, now 
supports a sub-set of the accounting functionality 
using the Cisco NetFlow-defined PDU formats. 
Riverstone Networks, formally a portion of Cabletron, 
has a protocol called LFAP: Lightweight Flow 
Accounting Protocol. Related software, such as slate, 
is available at [NMOps]. 


We are looking at the possibility of supporting 
flow processing based on the implementations of these 
or other vendors. Potentially one could build this sup- 
port for FlowScan by enhancing cflowd to support 
other flow export implementations, by modifying 
other collectors to produce the cflowd-defined raw 
flow file format, or by modifying FlowScan or the 
underlying Cflow perl module to be able to process a 
stream of flows of a different format. 


Availability 


FlowScan 1s freely available under the terms of 
the GNU General Public License [FlowScan, GPL]. 
FlowScan-1.002, that was released March 21, 2000, 
was used at approximately 100 sites. FlowScan-1.003 
was released September 15, 2000. More information 
on FlowScan is available at http://net.doit.wisc.edu/ 
~plonka/FlowScan/. A mailing list for flowscan users 
is archived at http://net.doit.wisc.edwplonka/list/flowscan/ . 


Likewise, RRGrapher is freely available under 
the same terms at http://net.doit.wisc.edu/plonka/ 
RRGrapher/. 


FlowScan’s primary building blocks, cflowd and 
RRDtool were both developed with the support of 
CAIDA. They are are freely available at http://www. 
caida.org/tools/measurement/cflowd/ and http://ee-staff. 
ethz.ch/“oetiker/webtools/rrdtool/, respectively. 


Summary 


FlowScan has become a useful tool for our net- 
work engineering team. We have run it nearly 
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continuously to measure and analyze all traffic to and 
from our campus since late 1998. The resulting reports 
and graphs have been of use for the detection of abuse 
and other anomalies that are detrimental to the net- 
work backbone’s performance as a whole. Without 
such a tool, these incidents would have gone unex- 
plained. Also, FlowScan’s long-term graphs have been 
used to convey the impact of Napster to our manage- 
ment, and to justify progressive upgrade and band- 
width acquisition plans for the campus. With its public 
release, FlowScan has proven useful for these and 
other purposes by a user base of educational institu- 
tions, government institutions, corporations, and Inter- 
net service providers. 


When attempting to meet the challenges of man- 
aging a heavily utilized IP network, near real-time 
traffic analysis and visualization quickly becomes an 
essential technology. One way to provide these capa- 
bilities is by utilizing Intemet traffic flow profiling 
based on technology available in most networking 
equipment. FlowScan is a system designed to provide 
this analysis continuously in near-real time and can be 
an effective tool to better understand Internet traffic. 
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ABSTRACT 


Most denial-of-service attacks are characterized by a flood of packets with random, 
apparently valid source addresses. These addresses are spoofed, created by a malicious program 
running on an unknown host, and carried by packets that bear no clues that could be used to 
determine their originating host. Identifying the source of such an attack requires tracing the 
packets back to the source hop by hop. Current approaches for tracing these attacks require the 
tedious continued attention and cooperation of each intermediate Internet Service Provider (ISP). 
This is not always easy given the world-wide scope of the Internet. 


We outline a technique for tracing spoofed packets back to their actual source host without 
relying on the cooperation of intervening ISPs. First, we map the paths from the victim to all 
possible networks. Next, we locate sources of network load, usually hosts or networks offering the 
UDP chargen service [5]. Finally, we work back through the tree, loading lines or router, observing 
changes in the rate of invading packets. These observations often allow us to eliminate all but a 
handful of networks that could be the source of the attacking packet stream. Our technique 
assumes that routes are largely symmetric, can be discovered, are fairly consistent, and the 
attacking packet stream arrives from a single source network. 


We have run some simple and single-blind tests on Lucent’s intranet, where our technique 
usually works, with better chances during busier network time periods; in several tests, we were 
able to determine the specific network containing the attacker. 


An attacker who is aware of our technique can easily thwart it, either by covering his traces 
on the attacking host, initiating a “‘whack-a-mole” attack from several sources, or using many 


sources. 


Introduction 


One of the major problems on the Internet today 
is denial of service (DoS) attacks against machines 
and networks. As opposed to other types of attacks, 
DoS attacks attempt to limit access to a machine or 
service instead of subverting the service itself. DoS 
attacks are simple to design and implement, and there 
is a plethora of readily available source code which 
will perform the task. DoS attacks send a stream of 
packets at a victim that swamps his network or pro- 
cessing capacity, denying access to his regular clients. 


There are two basic targets of DoS attacks: 
machines and networks. SYN attacks [11] are an 
example of an attack against a machine. In these 
attacks, a series of TCP SYN packets are sent to a 
host, filling its table of ‘‘half-open” TCP connections. 
Normal connection attempts are dropped. The basic 
problem with a skillfully run SYN attack is that the 
clients and the attackers are indistinguishable without 
further processing. The server must issue SYN/ACK 
packets and wait for the client to respond. This partic- 
ular attack can be mitigated with appropriate algo- 
rithms in the server [11]. Other machine attacks may 
be more difficult to defend against. 


The second target type, networks, are much more 
difficult to defend. Here, the goal is to overload a 
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company’s connection to its ISP. The attacker focuses 
a large stream of data towards the company’s network, 
often from a number of sites. The company’s connec- 
tion becomes congested, resulting in packet loss. Since 
routers cannot distinguish between attacking packets 
and valid client packets, they drop them with equal 
probability. If the attacker can send packets fast 
enough, the drop rate can become so high that an 
insufficient number of a client’s packets get through. 
Thus, clients cannot not get reasonable service from 
any machine beyond the loaded link. The most com- 
mon of this type of attack is the Smurf attack [8], 
although recent distributed denial of service attacks 
(DDoS) [9] have been of this flavor. 


The major advantage of DoS attacks is that it is 
quite difficult to determine the actual source of the 
attack. Since the attacker can basically put any packet 
on the local wire, the attacker creates packets whose 
source IP address is invalid and completely random. 
Thus, when the victim receives these packets, they are 
unable to determine the source. The current technique 
for tracing a packet stream back to the source requires 
cooperation of all the intervening ISPs. This is some- 
thing that is difficult to obtain, since the victim is 
rarely a customer of all of the ISPs between it and the 
attacker. The standard technique will be discussed in 
more detai! later. 
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We have developed a method to trace a steady 
stream of anonymous Internet packets back towards 
their source. The method does not rely on knowledge 
or cooperation from intervening ISPs along the path. 
In additions, tracing an attacking stream requires only 
a few minutes once the system is set up for a victim. 


Basic Technique 


We begin by creating a map of the routes from 
the victim to every network, using any known map- 
ping technology [1, 6, 7]. Then, starting with the clos- 
est router, we apply a brief burst of load to each link 
attached to it, using the UDP chargen service [5]. If 
the loaded link is a component of the path of the 
attacking stream, our induced load will perturb the 
attacking stream. Thus, if the stream is altered when 
we load a link, this link is probably along the path 
from the source host of the attack to the victim host. If 
the intensity of the stream is unperturbed by the load, 
it is unlikely that the stream of attacking packets is uti- 
lizing that link, so we do not need to examine the net- 
works “‘behind” that link. 


We continue working back through the network 
router by router, pruning branches that do not perturb 
the attack, as we try to narrow the attack source to one 
network, at which pomt we can shift to more standard 
traceback methods by contacting the entity which con- 
trols that network. 


Executing a trace effectively does require signifi- 
cant preparation in the way of data collection. We 
need to collect network data, as well as traceroutes 
from the victim to all possible networks. Due to asym- 
metric routes, naively, directional data must be col- 
lected and maintained by reverse traceroute servers or 
other means in order to have perfect data. We collect 
outbound paths and assume that the incoming paths 
are approximately the reverse of those paths. While 
this is not completely accurate, by collecting the paths 
to all networks, we can determine what links could be 
used on a path from a given network to the victim’s 
network, so this assumption does not cause as many 
inaccuracy as might otherwise occur. 


Because we need to induce isolated load on spe- 
cific network segments that are not in our purview, we 
must identify sources ‘willing to” (read: will) per- 
form that task. We recognize that ISPs are now quite 
regularly turning off the services that we exploit to 
induce these loads. Thus, we must identify cooperative 
hosts at the right places in our network map in order to 
do produce the required load. 


This element of the technique is worrisome, 
since it constitutes a brief denial-of-service attack on 
that network link. Hackers already employ bulk ver- 
sions of this approach for denial-of-service attacks. 
Our technique, on the other hand, carefully limits load 
to segments only long enough to rule them out as a 
possible component of the suspected path. The differ- 
ence is analogous to that between a sword and a 
scalpel. 
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In any case, we recognize the antisocial aspect of 
this technique, and expect that the tool will be used 
rarely and only in appropniate situations. Possible 
users include law enforcement, the military, ISPs, and 
companies policing their own private intranets. 


Before attacks or victims are even known, a 
trusted machine must develop and maintain a current 
database of networks and load generators. The current 
version of the tool executes the trace from the victim 
(targeted) network, but a sufficient complete map of 
the Internet might allow a neutral third party to run the 
detecting utility, which would allow flexibility in 
where to spread some of the bandwidth cost of the 
tool. 


In either case, the tracing machine emits packets 
that stimulate traffic flow through a desired router or 
link. A visual display of various statistics of the 
incoming packets on the victim’s network helps deter- 
mine if that link is used by the packets. 


An operator using a tool to probe links on the 
path back to the attacker. The application of load is 
done manually (see Figure 1). Though there are algo- 
rithms that might automate this process, we require 
human intervention to reduce the cost of programming 
errors. We try to supply the operator with information 
about the amount of load she 1s inflicting on networks, 
and she can chose to stop using packet-source net- 
works that have already generated a lot of load. 


If the induced load is sufficient to induce drops 
of incoming packets, it quickly and dramatically 
affects the attacking flow. The discomfort to ISPs and 
end users is brief enough that it likely to escape notice. 
If the load does not induce loss, it may be necessary to 
run the load generators longer and seek more subtle 
effects on the workload. 


Our technique appears to work better when the 
network is already heavily loaded, though one can 
imagine more subtle statistical effects that may be 
detectable when the Internet is relatively quiet. Our 
attempts to discover such effects has met with little 
success. We found we were interacting with cache and 
other optimizations in various routers. In some cases, 
our applied load actually increased the packet attack 
rate! 


Assumptions 


Our technique does rely on several assumptions, 
but our experience indicates they are often valid and 
the technique can work. 


Assumptions About the Internet 


We assume that most routes over the Internet are 
symmetric. Asymmetric routes confuse our mapping, 
traceback and loading. However, the proliferation of 
reverse traceroute servers, which has proven quite use- 
ful for network diagnosis and debugging, might also 
facilitate construction of at least a partial directional 
map of routes. 
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We also assume that we can generate enough 
load on a particular Intemet link to affect perfor- 
mance, in particular loss, statistics of the stream of 
attacking packets. We must have access to enough 
packet generators beyond the tested link to load it, 
which can be challenging across infrastructure with 
fast links and slower downstream networks. The tech- 
niques for doing this will be discussed below. 


Hacking Behavior 


We assume that the attack is from a single host, 
at a fairly consistent rate, and runs for a reasonably 
long time. Denial-of-service attacks are more vexing if 
they are ongoing, and we have seen attacks that last 
for weeks. We have seen attack rates of 200-500 pack- 
ets per second from a single host. We need time to 
move equipment and programs into place, map routes, 
and perform the actual traceback. 


Bizarre behavior can occur during the traceback, 
so we have to examine clues carefully. For example, 
the operator might notice an attacking stream drops by 
33% rather than dropping off entirely. Such behavior 
would be consistent with two or three concurrent 
attacks from separate hosts; it also possible that the 
attacking stream is being load-balanced across three 
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Figure 1: Screen shot of trace-back program. The left-hand screen gives information about the amount the usage of 
different hosts to generate pain. The bottom is a graph showing the number of packets received per second. The 
right-top shows the traceback step. The bottom of the traceback shows the path so far, and the top shows the 
possible next hops. The horizontal double lines are load-balanced lines, so these two IP addresses are really 
equivalent, for traceback purposes. 
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different links. Unfortunately, only one packet stream 
can be traced at a time, so being able to distinguish 
among the streams would be essential to be able to 
perform the trace. The operator might be able to use 
the arriving TTL value, assuming packets within each 
stream are launched with the same TTL value, and 
with each stream from different hop distances away. 
tcpdump’s filters provide the tools necessary to isolate 
such parameters, so that feature of the tool can be used 
if one of these parameters are sufficient to distinguish 
between streams. 


We assume the attacker does not know that her 
packets may be traced. An effective hacker attacks 
from co-opted hosts and never returns to the attacking 
machine. She hides her trail through a thread of login 
sessions across many hosts and networks before 
attacking the target. The dental-of-service attack we 
target with this tool is a one-way packet flow, which 
does not re ly on interactive login sessions. 


We assume that there is something forensically 
interesting at the source of the attack. The effort of 
running our tool may not be justified if the result is 
just disabling one attacking host or convincing one 
community of computers to enforce ingress filtering 
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[10]. We may be able to catch someone who was not 
very cautious because he did not expect his packets to 
be traced. The difficulty of the tracing task renders this 
a common assumption of hackers. 


We also assume that the attacker is unfamiliar 
with the techniques we provide here. These techniques 
are easily thwarted in several ways, including modify- 
ing the attacking program to vary the source of the 
attack, altering the frequency of the packets randomly, 
and attacking from many different sources (the 
““‘whack-a-mole”’ attack). 


Network Load: No Gain, No Pain 


Once we have determined the path to each net- 
work on the Internet, the traceback 1s done by walking 
backwards through the resulting directed graph. We 
load a link and hopefully cause enough packet-loss to 
see a noticeable drop in the rate of attacking packets. 
If a significant drop occurs, we can be fairly certain 
that the tested link is on the path from the attacker to 
the victim. Otherwise, either the link 1s not on the path 
or we did not provide enough load, or ‘pain,’ to that 
link to incur packet loss. Note that since most links are 
full duplex, we need to load the link in the direction 
towards the victim. 


This traceback requires making a high capacity 
link very busy for a short period of time, on the order 
of a second. It is difficult to generate a flow of packets 
from a single host that will do this: it would have to 
come from a fast host on a fast, unloaded link. We 
would prefer some leverage, some “gain,” on packets 
we emit. If we send out a flow of x bits per second 
(bps), we want the resulting flow across the link to be 
of kx bps, were k is large enough. 


To produce the load, we could send a series of 
messages, such as ICMP echo request (ping) packets 
[4], from the victim’s network out to distant networks 
whose retum path we expect to include the link we 
wish to load. However, using ICMP echo request 
packets gets us only one byte in retum for every byte 
we send out, which is a gain of only 1. In addition, 
the retum packets traverse the entire network back to 
the victim, which loads the entire set of links from the 
assistant network to the victim, which obscures the 
data when trying to determine the third link out. Send- 
ing ICMP echo requests from a separate network dedi- 
cated to this service is also problematic, since the 
nature of Internet routing means that it is hard to 
assure that their return path traverses the link we are 
testing. 


Instead of sending packets from the victim’s net- 
work, we send spoofed packets from a test host 
located elsewhere on the network. When testing a par- 
ticular link, we send probe packets to the router on the 
far end of the link, using as a return address the router 
on the near end of the link. The near router indignantly 
discards the unsolicited replies (if using TCP, it actu- 
ally may reset; for UDP, it may reply with a ICMP 
Port Unreachable). 
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More Gain 


Many routers make special efforts to put rate 
limits on handling of ICMP echo requests, since they 
are used so often. More importantly, the gain of 1 does 
not help us much anyway. Thus, we need to use a dif- 
ferent service in order to supply the load. 


The most obvious choice of service to employ is 
the forgotten tiny service TCP character generator 
(chargen) [5]. This service generates continuous data 
to anyone who connects to it, exactly what we want. 
The rate of data flow is limited in general by the rate 
that the data 1s acknowledged by the client machine. 
At the cost of a few TCP ACKs from our side, we can 
coax a steady stream of data out of a site supporting 
this service. Several of these routed over the target 
link will generate substantial load. We could even use 
the TCP ACKs to pulse all the transmitters to provide 
a fine burst of load by ACK-ing several open chargen 
sockets simultaneously. TCP chargen is tumed off on 
many of the Internet’s hosts and routers, but there are 
many that run the service, and they are easy to fine. 


We recognized two major problems: the TCP 
processing on our local host slows this chargen stream 
down more than we would like, and, more impor- 
tantly, the chargen stream still must traverse the path 
all the way back to the sender, unless we try TCP 
sequence guessing and IP spoofing, which gets very 
difficult very quickly. We can circumvent this second 
problem by using UDP chargen instead of TCP, and 
spoofing the packets, but this method provide little 
gain, as we usually get around 102 bytes back for our 
40 bytes, a gain of only 2.55. (We include 12 bytes of 
data in our packets that give information about the 
actual source of them.) The chargen RFC specifies 
that the return packet should have between 0 and 512 
bytes of data [5] (not counting the 28 bytes for the IP 
and UDP headers [2] [3]). We found, however, that 
some Windows NT 4.0 hosts violate this standard and 
return up to 6,000 bytes in response to a single packet, 
a gain of 150! 


A spoofed ICMP echo request to a broadcast 
address can yield gain as well. By locating networks 
‘beyond’ the link to send directed broadcast ICMP 
echo requests to, we get a gain of one for each host on 
that network which responds. Unfortunately, many 
routers process broadcast ICMP echo requests in such 
a way that only the router itself returns a packet. This 
is, of course, fortunate for the potential victims of 
broadcast ICMP echo request attacks, and is, in fact, 
recommended for that reason [8]. However, it limits 
broadcast ICMP echo request’s usefulness to us. 


Such routers do let other broadcast traffic 
through, however, and we found that we could obtain 
gains in excess of 200 quite often using broadcast 
UDP chargen packets, even on networks without NT 
4.0 hosts. Surprisingly, many networks within Lucent 
still respond to broadcast address 0 instead of 255, so 
we had to check both to determine the correct one for 
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each network. Figure 2 shows a distribution of net- 
works and their gain for Lucent’s intranet. Note that 
the networks with a gain of less than | have a gain of 
0, which means that they did not respond to broadcast 
UDP chargen at all. 
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Figure 3: Example of traceback step. Packets are sent 
to A and B, spoofed from R2, in order to initiate 
packet flows towards the victim. This causes 
Increases congestion along the R3-R2 link, which, 

if sufficient, will induce packet loss. 


When we initiate the load, the goal is to load one 
line or, maybe one router. We certainly do not want to 
load the entire path back to the victim. We prevent this 
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in two different ways. First, as mentioned above, we 
spoof the retum address of the UDP chargen packets 
to be the address of the router on the victim’s side of 
the link. Second, we utilize multiple UDP chargen 
hosts. To test a link, we select networks that reside 
behind the link, as seen from the victim (see Figure 3). 
In particular, we select networks that have hosts that 
respond to UDP chargen broadcast packets. We select 
a network for each outbond link from the far router of 
the line we are testing. This strategy focuses the load 
on the line under examination; the packets travel to the 
machine over different lines, hopefully not affecting 
each other significantly (again, Intemet routing 1s not 
Inconsistent with their having traversed a common 
link previously in the path, though it is unusual). The 
load is limited by the lines the load must traverse, the 
speed of the networks where the load is being gener- 
ated, or our ability to emit UDP chargen request pack- 
ets. 


The average gain seen in our experiments is 
around 133.8 within Lucent. One misconfigured net- 
work had a gain of several tens of thousands due to 
oddities in its configuration (see below). We can easily 
generate 2,500 40-byte packets per second, or 800 
kbps. To flood a 10Mbps Ethemet only requires a gain 
of 12.5. Figure 4 shows the necessary gains to load a 
variety of line types. In order to flood a backbone link, 
such as an OC-48 or OC-192, one needs gains in 
excess of 3,000, which is larger than all but one of the 
gains that we have seen. However, when loading back- 
bone links, we have help from the rest of the traffic 
that is traversing those links, so the actual amount of 
traffic required to start packet loss is much less than 
the number in the table. Also, we could increase the 
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Figure 2: Distribution of gains seen using the broadcast address for Lucent’s intranet. The network that generated a 


gain 43,509 and is excluded from this graph. 
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rate of outbound packets greatly by using multiple 
computers that connect to the Internet over different 
links. 


Gain Required 


Line Type 
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Figure 4: Required gains to load a variety of line 
types, assuming 800 kbps of emitted packets. 


Note that these numbers are a bit rough, since 
some of those 2,500 packets will most likely be 
dropped. Also, we could use 28 byte packets instead 
of 40 byte ones, but it is not clear that we could trans- 
mit them much more quickly. 


We have discussed only one possible technique 
for loading the actual line; another possibility is to 
load the router. Diverting packet flow by sending a 
message directly to a router is quite difficult, as Inter- 
net backbone routers ignore various ICMP messages 
to redirect or stifle packet flow. Most methods to load 
a router have to tackle its system configuration to limit 
return data flow. Router designs also typically have 
almost all forwarding handled by a simple machine 
that just delegates di fficult tasks to a higher layer. Less 
legitimate options, such as hijacking BGP sessions or 
breaking into the router itself are much too malicious 
to be seriously considered. 


There are other possibilities on ways to slow 
routers, however. One option is to ping flood the 
router, i.e., send it ICMP echo requests as fast as pos- 
sible. A similar alternative is to send the router a flood 
of packets whose Time to live (TTL) value expires at 
the desired hop along the path, or to transmit a stream 
of UDP packets to high ports to stimulate responding 
UDP port unreachables. Since most routers seem to 
rate-limit UDP port unreachable messages, we aban- 
doned this idea before testing it extensively. The other 
methods do not seem to have a major effect. 


Another idea is to spew packets at the router to 
try and upset its routing table. That is, find some sort 
of packet it responds regularly to (TTL exceeded, echo 
request) and send it a bunch of packets with random 
return addresses. Coping with the packets will require 
enough attention to unsettle the route table cache. In 
order to combat the incoming stream, it may be useful 
to pick a handful of sources and cycle through them. 
This approach has not shown much promise when 
used within Lucent, perhaps because many Lucent 
routers use only a single default route so forwarding 
cache state is not a resource issue. 
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Results 


We obtained logins on various hosts throughout 
Lucent’s intranet. We ran a non-privileged program 
named sendudp to generate a stream of packets back to 
a nonexistent host our local network. In most cases we 
could trace the packets back to the “attacking” build- 
ing. In many we could traceback to the individual Eth- 
ernet. 


Some links did not respond to our applied load. 
In some cases we had to go a hop beyond the non- 
responding links (all the links that are connected to a 
machine that are one hop away from where we had 
traced back to) in order to find a link which, when 
loaded, affected the packet flow. Sometimes, we could 
pick up enough of a signal from one of these next 
layer links that we could continue. It was a quite man- 
ual process, however, which could become difficult on 
untestable links with a large number of incident links. 


With two exceptions, corporate users appeared to 
be ignorant of our tests. If the mapping is subtle, and 
the load applied for short periods, users are unlikely to 
notice the performance hit, or dismiss it as normal net- 
work variability. 


Early on, we confined our testing to a few net- 
works, and the network administrator received enough 
complaints to notice our activities several times. Our 
subsequent tests appeared to be unnoticed, though in 
neither case did we attempt to hide our activities. 


On one network which we used to generate load, 
the broadcast UDP chargen packet initiated a broad- 
cast storm on their network. This network had a gain 
in the tens of thousands. Local users definitely noticed 
every time we used it, since it brought the network to 
a halt. Of the 2,000 networks in Lucent, only this one 
appeared to be unstable in this matter. It is unlikely 
that our probe packets would be detected on such a 
poorly-run network, which is likely to have frequent 
packet storms from other causes. The Internet likely 
has an even lower rate of misbehaving networks. 


Alternative Strategies 


This solution is not the only possible one to DoS 
attacks. Since DoS attacks rely on anonymity, a solu- 
tion must eliminate some anonymity of hosts. There 
are two basic methods to do this: ensure that sufficient 
spoofed packets are never transmitted over the Internet 
and developing a method for tracing back packets if 
necessary. The first two methods discussed below 
attempt to stop some of the spoofing, by ensuring that 
the at least the source IP address is on the same net- 
work as the actual source of the packets. The last three 
methods discuss alternative methods of tracing pack- 
ets. 


The problem with many of these is that they 
require universal deployment in order to work. If a 
couple ISPs opt to not follow the method, then the 
attacker can just launch the DoS from such a network. 
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Filter Return Addresses at the Source 


There are many ways to solve the problem of 
anonymous packets. The most desirable is to enforce 
correct source addresses at or near their source via a 
method called ingress filtering [10]. A company or 
university should block outgoing packets that do not 
have appropriate return addresses. ISPs should have 
similar filters for each of their customers. Many fire- 
walls do this as a matter of course. 


This solution is undoubtedly the right one. 
Anonymous packets have no place on the Internet. 
However, these filters do make life more complicated, 
and for large users behind slow routers they can even 
degrade performance. For network administrators, 
these filters are an additional administrative problem: 
one more thing to install, maintain, and get wrong. 
Several RFC’s recommend it as essential for any 
responsible participant in the global routing system. 
Most firewalls have the ability and capacity to per- 
form these checks. The source-based filtering may 
upset mobile networking methodologies. 


Filtering n Backbone Routers 


Routers at the core of the Internet, those running 
BGP4 and exchanging full Internet routing tables, 
inherently enforce proper destination addresses on 
packets, since the routing system 1s built around for- 
warding the packet toward the value of this field. The- 
oretically, routers could perform a similar check on the 
source address, 1.e., drop those with source addresses 
that are inconsistent with their incoming interface. 


Unfortunately, the verification is not nearly so 
simple, since a packet may come from more than one 
possible incoming interface, so routers would have to 
maintain a huge amount of state. Not only do routers 
not have spare memory resources to maintain this 
state, they do not have spare CPU resources to per- 
form the verification. In the midst of sustained for- 
warding rates of millions of packets per second, often 
operating quite near if not at their maximum capacity, 
router designers must optimize for speed. An addi- 
tional lookup of the source information would require 
similar optimization, and subsequent re-engineering of 
many routers, an expensive and unlikely scenario 
unless ISPs are willing to pay for it. 


One could imagine that legal fallout from a par- 
ticularly damaging attack might force this scenario, 
and some routers may emerge that support such func- 
tionality service without re-engineering. In general, 
however, the industry has long resisted source-based 
policy routing, and we do not expect a fundamental 
change in this mind-set in the short to medium term. 


Tracing by Hand 


The obvious ad hoc solution to finding a spoof- 
ing host is to trace packets back to their physical 
source manually. This is done by contacting an ISP 
and having them test each link to determine if a large 
number of packets are traversing that link destined for 
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the victim network. This is done is in a tree-like man- 
ner similar to ours, or at the access points to their net- 
works. There are two basic methods to do this, either 
examine the traffic flow across the link, or manually 
disconnect a link and see if it alters the packet flow 
(essentially what we attempt to do without physical 
access). 


This method requires significant cooperation and 
attention from intervening ISPs, which has proven a 
problem in past incidents. They may not have the pol- 
icy, inclination, time, expertise, or the instrumentation 
to help out. Test equipment may not be available for 
some locations or links within their network. For 
example, some Cisco routers have been known to 
crash if IP DEBUG is used under sufficiently heavy 
load. 


Further, the traces may be needed off-hours: the 
Panix attack started a little after five one Friday after- 
noon. It may be hard to find someone at any hour at 
the ISP who can handle the technical details. Some- 
times attacks are only solved because a victim hap- 
pens to be well-connected to admin-able friends at 
ISPs that are willing to help them out. 


This cooperation is very helpful, but selective, 
and slows the process down immensely. A quicker 
method would be extremely useful. 


Shutting Down a Router 


One could imagine sending a message to a router 
requesting that it drop all packets for a particular desti- 
nation for a second or so. This interruption would be 
long enough to detect a break in incoming packets, 
without noticeably affecting service. Implementing 
this feature would provide an obvious denial-of-ser- 
vice attack of its own. The router could require that 
requests be strongly authenticated, but there is no 
infrastructure present for such validation in the current 
Intemet. In self-defense, a router would have to do a 
similar rate-limiting as it does with UDP responses, 
rendering the feature useless for a significant attack. 
Given the ease that an attacker can hide her attack, it 
probably is not worth deploying such a service. 


Marking Packets with IP Addresses 


Another alternative is to place the IP address of 
all the routers that a packet goes through during its 
flight across the Internet. This has two obvious disad- 
vantages: it requires CPU time of the routers and it 
increases the size of packets, especially in the case of 
routing loops. Both of these could be reduced by hav- 
ing it mark only every | in 7 packets through a given 
interface. If n is small enough, a long enough attack 
would give you the complete list of routers along the 
path, if not their actual order. If m 1s chosen large 
enough, the additional router time and packet size 
increase would be negligible. In practice, one might 
want to randomly vary N to avoid possible problems 
with routers synchronizing. If is too small, than the 
attacker can insert packets into the network that can 
‘fool’? your system into misdiagnosing the path. In 
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practice, you may want to keep only one address in a 
packet at a time in order to simplify the header. 


Ethics 


We acknowledge that our methods to traceback 
anonymous packets resemble techniques used by 
hackers. There are several questions to deal with in 
this area. 

1. Does the tracking attempt cause more damage 
than the actual packets? Obviously if the 
answer 1s yes, then we should not pursue the 
technique. We cannot provide a_ universal 
answer to this question; it really depends on the 
situation. If the anonymous packet stream has 
shut down the daytime service on your web 
server, it 1s perhaps not costing you enough to 
take serious action. If they are crashing your 
network and denying your customers access to 
the service you sell them, then perhaps stopping 
is worth the cost of congesting a few network 
links for a few seconds (it almost certainly 
seems so to you). The user of this method will 
have to make this judgment. 

2. Does leaving a service (such as UDP chargen) 
enabled on your machine implicitly mean you 
have given permission to use it? Our method 
does not attempt to gain access to private infor- 
mation or crash individual machines, but it does 
leverage accessible services from _ private 
machines. However, these machines have left 
the UDP chargen enabled (or whatever service 
1s employed). 

The easy answer is yes, but it runs dangerously 

close to the hacker’s defense that running a ser- 

vice with possible security holes indemnifies 
those who intentionally exploit it. On the other 
hand, we are not really exploiting a security in 
an implementation. Indeed, we are following 
the intended protocol specification exactly. 

Nonetheless, the essence of our tool is the 

imposition of a  denital-of-service of the 

attacker’s own denial-of-service attack against 
us. 


After much consideration, we must conclude that 
the appropriate answer to this question comes down to 
motivation. While a hacker is generally trying to harm 
the machine, gain access to private information, or 
journey on an ego trip, our tool is leveraging the 
machine for a secondary purpose that is helpful to the 
Internet community. 


We recognize that this argument may not be suf- 
ficient for some organizations that reside on the Inter- 
net. 


Discussion 


This technique is not ideal, either in efficiency, 
speed, or impact on other Internet users. We have 
shown that it works on an intranet, which tends to be a 
more controlled environment than the Internet itself. 
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It would be preferable to find a better solution 
involving ISP coordination and cooperation. Unfortu- 
nately, we have to admit that sometimes the perpetra- 
tors are at ISPs, so an official mechanism that tips 
them off might be completely impotent. We expect 
that ISPs will drift toward better solutions as their own 
clients demand assistance. 
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ABSTRACT 


In this paper we present an analysis of Shaft, an example of malware used in distributed 
denial of service (DDoS) attacks. This relatively recent occurrence combines well-known denial of 
service attacks (such as TCP SYN flood, smurf, and UDP flood) with a distributed and 
coordinated approach to create a powerful program, capable of slowing network communications 
to a grinding halt. 


Denial of service attack programs, root kits, and network sniffers have been around in the 
computer underground for a very long time. They have not gained nearly the same level of 
attention by the general public as did the Morris Jnternet Worm of 1988, but have slowly 
progressed in their development. As more and more systems have come to be required for 
business, research, education, the basic functioning of government, and now entertainment and 
commerce from people’s homes, the increasingly large number of vulnerable systems has 
converged with the development of these tools to create a situation that resulted in distributed 
denial of service attacks that took down the largest e-commerce and media sites on the Internet. 


In contrast, we provide a comparative analysis of several distributed denial of service tools 
(e.g., Trinoo, TFN, Stacheldraht, and Mstream), look at emerging countermeasures against some 
of these tools. We look at practical examples of these techniques, provide some examples from test 


environments and finally talk about future trends of these distributed tools. 


Introduction 


Network-based attacks are nothing new, but up 
to last year the techniques utilized were focused on 
simple point-to-point denial of service. By denial of 
service we mean overwhelming the victim host or net- 
work to the point of unresponsiveness to the legitimate 
user. We provide a little overview, by no means com- 
plete, of previous point-to-point denial of service tech- 
niques. There are four major point-to-point tech- 
niques: TCP SYN flooding, UDP flooding, ICMP 
flooding, and Smurf attacks. The first one misbehaves 
from the standard three-way TCP handshake causing 
resource consumption and bandwidth consumption, 
whereas the remaining ones intend to consume the 
victim’s bandwidth. 


The year 1999 saw an emergence of new denial 
of service tools. The change was inevitable: the 
growth of network pipes made simple point-to-point 
tools either useless, or the improved tracking capabili- 
ties easily shut down the source of the problem. Even 
though some solutions or at least containment methods 
exist for the above, the distributed variants as an evo- 
lution of coordinated many-to-one attacks escape the 
traditional model sufficiently. Rather than relying on a 
single source, attackers could now take advantage of 
some hundred, thousand, even ten thousand or more 
systems to inflict denial of service onto their victims. 
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Analysis 


The DDoS Network Model 


A Distributed Denial of Service Network follows 
a hierarchical model, with one or more attackers con- 
trolling a so-called handler, which in turn controls the 
hordes of agents that execute the commands relayed to 
them. 


The communication between the attacker and the 
handler, and between the handler and the agents is 
referred to as the control traffic of the network, 
whereas the communication between the agents and 
the victims is referred to as the flood traffic. Control 
traffic can be TCP, UDP, ICMP, or a combination of 
the three. Flood traffic consists of traffic generated by 
each individual point-to-point denial of service tech- 
nique, or sometimes a combination thereof. 


In order to remove himself from view, the 
attacker introduces additional layers between the vic- 
tim host(s) and himself. He can access the handler via 
a variety of mechanisms, the most popular being a 
simple te/net. More sophisticated tools use, or can take 
advantage of, more advanced techniques, such as 
encrypted TCP connections (ssh is a possibility for 
TFN, blowfish-encrypted proprietary as in Stachel- 
draht) or non-standard methods such as embedding 
commands in ICMP or UDP packets (e.g., LOKI [22, 
23] or Q). Additional care is taken to protect the han- 
dler, as it is the key control point and effectively the 
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anonymizer of the network. So as to eliminate a single 
point of failure, more than one handler is found in 
practice, and in most cases each handler has equal 
power over its agents. 


The agents are controlled from the handler, often 
using a different protocol than the one in effect 
between attacker and handler. It 1s speculated that this 
is done to evade correlation. The communication 1s 
not necessarily bidirectional, as there have been cases 
of oblivious transfers. Instructed by the attacker, the 
handler can control the numerous agents to perform 
the attacks by proxy. As we will see, some DDoS tools 
provide a clear overview of the DDoS network, e.g., 
enabling to determine the status and performance of 
each individual agent. 






Attacker 


Figure 1: A typical DDoS network 


Findings 

Shaft was initially detected through anomalous 
network activity. With the help of the network ana- 
lyzer Argus [1], spikes (see Figure 2) in the packet 
flows led to the discovery of the shaftnode agent on 
the compromised system within the local network. It 
was one of about 100 nodes in a Shaft DDoS network. 
The successful retrieval of an attack binary, the shaftn- 
ode agent, and eventually its source, permitted a thor- 
ough analysis of its functionality. 


Since the shaftmaster handler was not retrieved 
until four months later, it took simulation, thorough 
analysis of Argus [1] logs and a pinch of creativity to 
reconstruct the functionality of this attack tool. Simu- 
lation and analysis tools such as the Unix debugging 
tool strace, and disassembly of binaries are the main 
contributors to the understanding of Shaft. 


Communication Features 


As a first step, it was important to identify the 
network communication aspect of Shaft. Shaft (in the 
analyzed version, 1.72) 1s modeled after Trinoo [11], 
in that communication between handlers and agents is 
achieved using the unreliable IP protocol UDP. See 
Stevens [29] for an extensive discussion of the TCP 
and UDP protocols. Remote control is established via 
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a simple telnet connection to the handler. Shaft uses 
tickets for keeping track of the transactions issued to 
its individual agents. Both passwords and ticket num- 
bers have to match for the agent to execute the 
request. A simple letter-shifting (Caesar cipher, see 
Schneier [27]) is in use. 


Command Structure 


Next, analyzing the command structure of the 
tool provided additional understanding of the capabili- 
ties of Shaft. Using available source and the simple 
Unix command strings, we established the command 
syntax of both the agent and the handler. It provided 
insight into the capabilities of the handler that was not 
apparent from the agent source. A full listing of both 
the agent and handler commands can be found in 
Appendix 1 and 2, respectively. 
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Figure 2: Spikes in network activity. 


Detection 
Brief Description of Installation Methods 


As with previous DDoS tools, the methods used 
to install the handler/agent will be the same as 
installing any program on a compromised Unix sys- 
tem, with all the standard options for concealing the 
programs and files (e.g., use of hidden directories, root 
kits, kernel modules, etc.). The reader is referred to 
Dittrich’s Trinoo analysis [11] for a description of pos- 
sible installation methods of this type of tool. 


Further findings [32] have revealed that the Shaft 
DDoS tools were indeed used in conjunction with a 
root kit, an inetd-based (inetd is the Unix server that 
handles most incoming connections such as telnet and 
ftp) trojan, a trojaned secure shell (SSH) daemon, and 
a set of Unix shell scripts to automatically distribute 
the tools out to the individual agent systems. The pre- 
sent inetd-based trojan has been known to exist in the 
wild as early as May 1999. 


The distribution Unix shell script (from [32]), as 
sent with netcat [19] to the trojaned system, 1s as fol- 
lows: 

#1 /bin/sh 

echo "“oirdHFt" 
echor "QULT" 
Sleep 5 

echo: ed. tmp" 
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Sleep 5 

echo "rep user@host:shaftnode ./" 
sleep 5 

echo "chmod +x shaftnode" 

sleep 5 

acho. "./shattnode" 

echo "exit" 


This shell script installs the shaftnode agent on the 
system, by performing a remote copy from a reposi- 
tory host into the /tmp directory, making it executable 
and launching it. The reader is referred to [32] for a 
complete discussion of the installation, trojaning and 
rootkit-ing of the handler host. 


Algorithmic Overview of Attacks 


Upon launch, the shaft agent (the “shaftnode’’) 
reports back to its default handler (its ‘‘shaftmaster’’) 
by sending a “‘new <upshifted password>’’ command, 
which registers the new agent in the pool of agents 
available to the handler. For the default password of 
“shift” found in the analyzed code, this would be 
““tijgu’’. Therefore a new agent would send out “new 
tijgu’’, and all subsequent messages would carry that 
password in it. Only in one case does the agent shift in 
the opposite direction for one particular command, 
e.g., ““pktres rghes”. While it was initially unclear 
whether this was a mistake, a more thorough analysis 
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of the shaftmaster revealed that both shifts were used 
in an attempt to evade analysis. 


Incoming commands arrive as space separated 
items: command, upshifted password, command argu- 
ment, socket number, ticket, and optional arguments, 
which can be represented as the message flow diagram 
between handler H and agent A: 

1, A >H: "new", f(password) 
2. H>A-: cmd, f(password), [args], Na, Nb 
3. A > H: cmdrep, f(password), Na, Nb, [args] 
4. Jump to step 2. 
e f(X) is the Caesar cipher function on X 
e Na, Nb are numbers (tickets, socket nu mbers) 
e cmd, cmdrep are commands and command 
acknowledgments 
° args are command arguments 


The flooding occurs in bursts of 100 packets per 
host, with the source port and source address random- 
ized. This number is hard-coded, but it is believed that 
more flexibility can be added. Whereas the source port 
spoofing only works if the agent is running as a root 
privileged process, the author has added provisions for 
packet flooding using the UDP protocol and with the 
correct source address in the case the process is run- 
ning as a simple user process. It is noteworthy that the 
random function is not properly seeded, which may 


Time Protocol Src IP/Port Flow _ Dst IP/Port 

OH PM) tcp Z.Z.Z.Z.53982 = XXXX.21] 
239-32 tcp x.xX.x.X.1023 > y-y-y-y.514 
21:39:56 udp X.X.X.X.33198 > Z.Z.Z.Z.20433 
21:45:20 udp Z.Z.Z.Z.1765 > Xx X.X.X.18753 
21:45:20 udp X.X.X.X.33199 > Z.Z.Z.Z.20433 
21:45:59 udp Z.Z.Z.Z. 1866 > X.x.x.x.18753 
21:45:59 udp x.x.x X.33200 > ZZ.2.Z.20433 
21:45:59 udp Z.Z.Z.Z. 1968 > x.x.x.X.18753 
21:45:59 udp Z.Z.Z.Z.1046 > x.x.X.X.18753 
21:45:59 udp Z.Z.Z.Z.1147 => x.x.X.xX. 18753 
21:45:59 udp Z.Z.Z.Z.1248 > X.x.xX.x.18753 
21:45:59 udp Z.Z.Z.2.1451 > X.X.x.xX.18753 
21:46:00 udp x.x.X X.33201 > ZZ.Z.Z.20433 
21:46:00 udp X.X.X X.33202 > Z.Z2Z.2.20433 
21:46:01 udp X.X.xX.33203 > Z.Z.Z.Z.20433 
21:48:37 udp Z.Z.Z.Z.1037 > x.x.X.xX. 18753 
21:48:37 udp Z.Z.2.Z.1239 > X.x.x.x. 18753 
21:48:37 udp Z.Z.Z.Z. 1340 > x.x.X.x. 18753 
21:48:37 udp Z.Z.Z.Z.1442 > X.X.X.x.18753 
21:48:38 udp x.X.x.X.33204 > ZZ.2Z.2.20433 
21:48:38 udp x.X.X.X.33205 > Z.Z.Z.Z.20433 
21:48:38 udp x.X.xX.X.33206 > ZZ.Z.Z.20433 
21:48:56 udp Z.Z.Z.Z.1644 > x X.xX.18753 
21:48:56 udp X.x.XX.33207 = ZZ.Z.Z.20433 
21:49:59 udp X.X.X.x.33208 > Z.Z.Z.Z.20433 
21:50:00 udp X.X.x.X.33209 > Z.Z.Z.Z.20433 
21:50:14 udp Z.Z.Z.Z.1747 > x.x.xX.18753 
21:50:14 udp X.X.X.X.33210 > Z.Z.2.Z.20433 
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Table 1: Compromise flow on Nov 28. 
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lead to predictable source port sequences and source 
host IP sequences. 


The source port is generated with (R mod 
(65535-1024)+1024) where R is the output of the 
rand() function. This will generate source ports greater 
than 1024 at all times. 


The source IP is of the form R1.R2.R3.R4 where 
Rl, R2, R3, R4 are the outputs of rand() mod 255. The 
source IP numbers can (and will) contain a zero in the 
leading octet. 


Additionally, the sequence number for all TCP 
packets is fixed, namely 0x28374839, which helps 
with respect to detection at the network level. The 
ACK and URGENT flags are randomly set, except on 
some platforms. Destination ports for TCP and UDP 
packet floods are randomized. 


The client must choose the duration (“‘time’’), 
size of packets, and type of packet flooding directed at 
the victim hosts. Each set of hosts has its own dura- 
tion, which gets divided evenly across all hosts. This 
is unlike TFN [6] which forks an individual process 
for each victim host. For the type, the client can select 
UDP, TCP SYN, ICMP packet flooding, or the combi- 
nation of all three. Even though there is potential for 
having a different type and packet size for each set of 
victim hosts, this feature is not exploited in this ver- 
sion. 


When a general command is issued, it is sent to 
all hosts listed in a hidden file containing all the Shaft 
agents, in genera] with a timeout of 30 seconds. To 
date, no mechanism to alter that timeout has been 
found. Some commands have longer timeouts, up to 
300 seconds. A list of outstanding tickets (transactions 
waiting to complete) is available to the attacker with 
the “‘ltic’? command, which lists the ticket number and 
its corresponding remaining time. The attacker can 
visually correlate the ticket number to the actual com- 
mand by scrolling back in his screen buffer and com- 
paring the number that was printed after the execution 
of the command, similar to seeing a process id dis- 
played when sending a process into the background on 
a Unix system. 


The author of Shaft seems to have a particular 
interest in statistics, namely packet generation rates of 
its individual agents. The statistics on packet genera- 
tion rates are possibly used to determine the “yield” 
of the DDoS network as a whole. This would allow 
the attacker to stop adding hosts to the attack network 
when it reached the necessary size to overwhelm the 
victim network, and to know when it 1s necessary to 
add more agents to compensate for loss of agents due 
to attrition during an attack (as the agent systems are 
identified and taken off-line). 


Packet Flow Analysis 


In this section we will look at a practical exam- 
ple of an attack carried out with the Shaft distributed 
denial of service attack tool, as seen from the attack- 
ing network perspective. 
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The handler is listening on port 20433, and an 
existing connection on port 20432 is awaiting the 
commands of the attacker. The packet flow is illus- 
trated in Table 1. 


There is quite some activity between the handler 
and the agent, as they go through the command 
request and acknowledgement phases. There was also 
what appeared to be testing of the impact on the local 
network itself with, among others, UDP packet flood- 
ing against the broadcast address (first 2-3 spikes), fol- 
lowed by ICMP flooding as shown in Figure 2. See 
Figure 3 for a fine-grained view. 
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Figure 3: 28 Nov 1999 floods 21:00-23:00. 


The interesting portion is the first three lines. It shows 


the penetration from the handler (z.z.z.z) using the 
inetd-based trojan with source port 53982 and destina- 
tion port 21 (any inetd related port would have 
worked), the download of the shaftnode binary from 
y.y.y.y vla rcp (remote copy, port 514), and the regis- 
tration of the shaftnode agent with its shaftmaster han- 
dler. The theory that these were the traces of the pene- 
tration was confirmed by findings [32] on the handler 
host. The ten second delay between the packet on port 
21 and the remote copy on port 514 is consistent with 
the script mentioned in the section on installation 
methods. Later that night, the attacker performed sev- 
eral attacks (three UDP and one _ combination 
TCP/UDP/ICMP) in order to test the Shaft network 
further, as illustrated in Figure 4. Let us look at the 
individual phases from a later attack after it became 
possible to record the packet contents, as well as gen- 
eral flow data, subsequent to a determination of the 
agent, handler and communication ports. Subse- 
quently, the handler continued to send such packets 
even though the agent had been disabled and the host 
integrity recovered. This is illustrated in Table 2. 
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specified, which is again confirmed. The last line indi- 
cates the type of attack, in this case “‘the works’’, 1.e., 
UDP, TCP SYN and ICMP packet flooding combined. 
Failure to specify the type would make the agent 
default to UDP packet flooding. 


Now the list of hosts to attack and which ones 
they want statistics from on completion, as shown in 
Table 3. To protect the identity of the victims, the 
hosts IP number have been replaced with aa.a.a 
through j,j,j,. 


Packet flows 


20000 | 


10000 + 


Table 2: Setup and configuration phase on Dec 4. 


time flow command 

18:09:24 Z>xX ~~ owntiygu a.a.a.a 5 5256 
18:09:24 Xh%—>Z _— owning tiygu 5 5256 a.a.a.a 
18:09:24 Z—>xX _ pktres tijgua.a.a.a 5 1993 
18:09:24 Z—+>X ~~ owntigub.b.b.b 5 78 
18:09:24 Z—>X _  pktres tijgu j.j.j.j 5 8845 
18:09:24 Z>X ~ owntiguc.c.c.c 5 6247 
18:09:25 Z>xX  owntygud.d.d.d5 4190 
18:09:25 Z>X  owntiygu e.e.e.e 5 2376 
18:09:25. X—>Z _— owning tygu 5 78 b.b.b.b 
18:09:26 X—>Z _ — owning tigu 5 6247 c.c.c.c 
18:09:27 X—>Z _~ owning tygu 5 4190 d.d.d.d 
18:09:28 X—>Z _~ owning tygu 5 2376 e.e.e.e 
18:21:04 XZ __ pktresrghes 5 1993 51600 
18:21:04 XZ __ pktres rghes 0 0 51400 
18:21:07 XZ __ pktres rghes 0 051500 
18:21:07 X-—>Z__ pktres rghes 0 0 51400 
18:21:07 X-—>Z__ pktres rghes 0 051400 





51 


101 





Time (Minutes) 


Figure 4: Further testing 29 Nov 1999 02:00-07:00. 
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time flow command 

18:06:40 Z->X _ alive tygu hi 5 8170 
18:09:14 Z—>X _ time tijgu 700 5 6437 
18:09:14 X->Z _ time tygu 5 6437 700 
18:09:16 Z—>X _ size tijgu 4096 5 8717 
18:09:16 X—>Z © sizetigu5 8717 4096 
18:09:23 Z>X _ type tiygu2 5 9003 


time flow command 

18:24:25 Z>xX  owntiygue.e.e.e 5 4493 
18:25:53 Z—>xX — owntiygu b.b.b.b 5 9392 
18:27:05 Z—>xX — own tigu a.a.a.a 5 3085 
18:27:06 X—>Z _— owning tiygu 5 3085 a.a.a.a 
18:33:52 Z>X  owntiyguc.c.c.c5 1878 
18:33:53 X—>Z _— owning tygu 5 1878 c.c.c.c 
18:36:04 X—>Z __ pktresrghes 00 104100 
18:36:20 Z—>X _  pktres tigua.aaa5 1511 
18:36:21 XZ _— owning tygu 5 1754 a.a.a.a 
18:37:33 X—>Z__ pktresrghes 0 0 81700 
18:38:13 Z>xX ~~ owntygu f.ff.f5 3126 
18:38:13. Z—>X _ _ pktres tiygu f.f-f.f 5 4697 
18:38:14 X—>Z _ — owning tiygu 5 3126 f£.f.f.f 
18:38:47 X—>Z__ pktresrghes 5 1511 76600 
18:39:15 Z—>X  owntijgu g.g.g.g 5 4272 
18:39:16 X—>Z _— owning tygu 5 4272 g.g.¢.¢ 
18:39:41 Z>xX ~ owntiyguc.c.c.c 5 8850 
18:39:41 Z—>xX _ - pktres tijguc.c.c.c 5 9924 
18:40:43 Z—>X — owntiguc.c.c.c 5 2672 
18:41:25 X—>Z _— owning tiygu 55195 h.h.h.h 
18:45:33 XZ __ pktresrghes 5 9924 53700 
18:48:01 X—>Z__ pktresrghes 00 48800 
18:49:54 X-—>Z_ _ pktres rghes 5 4697 45700 
18:50:56 X—>Z __ pktresrghes 0 0 44900 
18:51:22 X—~>Z__ pktresrghes 00 45700 
18:53:04 X—>Z __ pktresrghes 0 0 63700 
18:54:47 Z>xX — owntiygui.11.15 2086 
18:54:47 Z—>X - pktres tiygu 1.11.15 6980 
18:54:47 X>Z _— owning tygu 5 2086 1.1.1.1 
19:06:27 X-—>Z__ pktresrghes 5 6980 241200 


Table 4: More hosts and statistics. 


Now that all other parameters are set, the handler 


Table 3: Host list and statistics. 


The handler issues an “alive”? command, and 
says “‘hi” to its agent, assigning a socket number of 
5”? and a ticket number of 8170. We will see that this 
““socket number” will persist throughout this attack. A 
time period of 700 seconds is assigned to the agent, 
which is acknowledged. A packet size of 4096 bytes is 


2000 LISA XIV — December 3-8, 2000 — New Orleans, LA 


issues several ““own” commands, in effect specifying 
the victim hosts. Those commands are acknowledged 
by the agent with an “owning” reply. The flooding 
occurs as soon as the first victim host gets added. The 
handler also requests packet statistics from the agents 
for certain victim hosts (e.g., “‘pktres tijgu a.a.a.a 5 
1993’’). Note that the reply comes back with the same 
identifiers (“5 1993”) at the end of the 700 second 
packet flood, indicating that 51600 sets of packets 
were sent. One should realize that, if successful, this 
means 51600 x 3 packets due to the configuration of 
all three (UDP, TCP, and ICMP) types of packets. In 
turn, this results in roughly 220 4096 byte packets per 
second per host, or about 900 kilobytes per second per 
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victim host from this agent alone, about 4.5 megabytes 
per second total for this little exercise. A graphical 
view can be seen in the first portion (minutes | 
through 12) of Figure 5. 


Note the reverse shift (“‘shift”” becomes “‘rghes’’, 
rather than “‘tijgu’’) for the password on the packet 
statistics. Continuing on with the attack, as shown in 
Table 4, the attacker selects new targets in a staggered 
manner, but still keeping the established settings of the 
700 second combination type attack of 4096 byte 
packets. The yields of the attack vary from roughly 
800 kilobytes per second per host in a multi-target set- 
ting to 4.2 megabytes per second per host in a single 
target setting (Target I). The staggered approach can 
be observed in the right two thirds of Figure 5 (min- 
utes 14 through 50). 

Cryptographic Aspects 

Shaft incorporated several noteworthy tech- 
niques for keeping information secret. For one, the 
letter-shifting or Caesar cipher, was applied several 
times within this tool. As described previously, the 
transaction password “‘shift’’ was shifted by one letter 
upwards to generate the string “‘tijgu’’ observed on the 
network. However, a different shift in the opposite 
direction generated “‘rghes”’ for the return statistics. 
While the author(s) of this program did not encrypt the 
entire message exchange between handler and agent, 
they did nevertheless obfuscate the real strings, such 
as applying the shift to the handler IP numbers in the 
binary and also the port numbers in the case of a 
‘““switch’? command, namely by adding an offset to the 
real port number. 
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As with the original Trinoo tool, the Shaft han- 
dler contained 13-character strings, strangely resem- 
bling Unix crypt() output. Through close analysis of 
the handler code, it was established that they repre- 
sented the access passwords to the control port of the 
program, that is where the attacker would connect to 
and perforin the distnbuted denial of service from a 
convenient, but not quite menu-driven, command line. 
The actual passwords were recovered in a similar 
fashion to the ones from Trinoo and Stacheldraht, 
except that the above shifts had to be performed on the 
ciphertext first. 


Similar to the handler settings in earlier tools, the 
author of Shaft attempts to keep the list of its agents in 
a non-trivial format. Other tools encrypt that list using 
Blowfish, but this tool packs the four octets of the IP 
number into a 4-byte integer and writes the ASCII rep- 
resentation of that number to a file, one per line. For 
example, adding the agent with IP address 127.0.0.1 
using ‘+node 127.0.0.1”’ would yield a line contain- 
ing “16777343”, which 1s: 

127 x 256° + 0x 256' +0 x 2567 + 1 x 256°. 
In order to extract the IP numbers from the list, one 
would apply the reverse transformation. 


Anomaly Detection 


The network flooding which took place was ini- 
tially noticed after a cursory glance at the hourly net- 
work flow data files recorded using an Argus [1] mon- 
itor at the main Internet connection point. Without any 
such monitoring the activities would have almost cer- 
tainly gone unnoticed since the floods took place over 
the Sunday-Monday night. Other IDS records 


Target A —--— Target B 
Target C Target D 
=== Target E.-=—~= Target. F 
=== - Target. G Target H 
Target | 


26 31 


Time (approx minutes) 





Figure 5: 4 Dec 1999 floods. 
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indicated a possible UDP portscan had taken place but 
for host IPs not possible on the local net blocks. 


For example, the typical argus data file at that 
time of day (night) would be 4Mbytes but these grew 
to between 40 and 100Mbytes. Analyzing such large 
data files takes significant effort and resources (hence 
the hourly rotation) but it was possible to determine 
the start time of the rapid rise in connections and then 
tracing the external connection (handler) and internal 
flooder (agent) becomes a matter for trial and error 
and some measure of good fortune. In this instance the 
first guess was used to contact the host administrator 
who was able to locate the process still left running 
(although inactive), obtain an lsof [18] output and 
recover residual files and logs. 


Reducing the data for the hourly flows down to 
something suitable for graphical display was compli- 
cated by the very large number of data points which 
overwhelms most of the standard graphing or statisti- 
cal packages. 


Other traffic monitoring applications which 
might have indicated that an unusually large network 
flow had occurred would not usually have an accurate 
time nor could have been used to trace the external > 
internal communications channel correspondence. For 
example snmp monitoring of packet numbers 1s a pop- 
ular method. Accumulated byte counts per sub-net 
(host, port, etc) could also have been obtained using 
NeTraMet [5] but again this would have been inade- 
quate for the post event analysis. 


Impact of Victim Hosts/networks 


The effect of the combined outpouring of packets 
has already been considered from the point of view of 
the target victim but it should be noted that very little 
legitimate traffic was able to move over the Internet 
gateway while the flood took place. This secondary 
denial of service would be of major significance dur- 
ing normal working hours and when combined with 
several such agents distributed around one site can 
lead to saturation of the essential backbone infrastruc- 
ture and routers. 


Impact on network — given the time of day it had 
little or no impact apart from slowing external port 
scanners (!) — maybe it is worth noting that on typical 
asynchronous ATM external connections there would 
be an impact on outgoing verus incoming. The impact 
of one host running flat out will be a lot less than sev- 
eral hosts running as agents. Deliberately limiting 
agents to one per site would have considerable bene- 
fits when it comes to avoiding detections while still 
retaining effective DoS of the target(s). 


Secondary Effects 


Poor DNS response (if any) — even problems 
managing network devices during the peaks. The 
flurry of “response” packets caused by the floods can 
create additional complications within the network. 
Due to the “inband-signalling”’ nature of TCP/IP, the 
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control messages related to network management must 
travel over the very same congested network. 


NIDS vs. Active Scanning 
Network Intrusion Detection Systems (NIDS) 


During very high (near saturation) flows almost 
no event of any kind would be logged by an IDS sys- 
tem — they would either have to drop packets at a very 
high rate or require multi-CPU architectures in order 
to combine packet collection and packet state analysis. 
AS pipe capacities continue to grow (Gigabit, etc) 
there will be serious difficulties for network flow 
monitors such as Argus to keep up based on the typ1- 
cal PC architecture (there is seldom a budget available 
for a top end machine which will, hopefully, be wast- 
ing cycles 99 % of the time waiting for such events). 


Passive Scanning 


For the purpose of detecting malicious activity, 
certain features of the whole DDoS package have to 
be considered and provide clues for passive scanning 
of such events. This program does not provide for 
code updates (like TFN or Stacheldraht). This may 
imply “rcp” or “ftp” connections during the initial 
intrusion phase (see also [11]). As found in [32], the 
intruders used “‘rcp”’ in their distribution scripts, but 
this could easily be altered. 


The program uses UDP traffic for its communi- 
cation between the handlers and the agents. Consider- 
ing that the traffic is not encrypted, it can easily be 
detected based on certain keywords. Performing an 
““ngrep”’ [20] for the keywords mentioned in the syn- 
tax sections (Appendix 1 and 2), will locate the con- 
trol traffic, and looking for TCP packets with 
sequence numbers of 0x28374839 (decimal 
674711609) may locate the TCP SYN packet flood 
traffic. The latter traffic can be detected through its 
secondary effect of causing SYN|ACK and RST|ACK 
traffic with sequence numbers of 0x2837483a (deci- 
mal 674711610), as pointed out by Richard Bejtlich 
(who has been witnessing these effects — with this 
same sequence number — for well over a year [3]). 
Source ports of the flood traffic are always above 
1024, and source IP numbers can include zeroes in the 
leading octet. 


Strings in this control traffic can be detected with 
the “‘ngrep”’ program using the same technique shown 
in [11, 12, 13]. Here are some examples that will 
locate the control traffic: between the handler and the 
agent, independently of the port number used. 


# ngrep -i -x “alive tijgu" udp 


Volo? Oss lO. aOOl o> O92 os ehOe 22a 753 


bill sO 169 “7.6: S65. 20> 7469 alive ti 
Ga 07.75, 207016 69" 20-55 Hou id": 
20% 3893. 6¢-34. 345 0a 8641. 


We O26 Sul 2a Oe Or POD LOO Owl 20483 


616069). 7:6:56.5, 220 “7 4:°69 alive ti 
Oa OO/. 25.20.35: 2033-36 jgu 3 66 
38° Si" 2062:76e. 61268 41 blah 
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The above will show the ‘‘alive’’ messages exchanged recv udp sport=20433 
between handler and agents. data="alive" nmatch=1 
## ngrep -i -x "pktres|pktstat" udp end shaft 
Uo 192.966 310.7214995 -07197, 166 710. 120432 
aveats 
70 6b 74 73 74 61 74 20 ~= pkt stat oe | | | 
Fle-69. 6a 6970 15 20 356 20 tijgu 5 It should be emphasized again and again that the 
S30 35-97- 20-30 195720 passive and active detection triggers rely on “old” 


numbers, strings, etc. and that they are often trivial to 
modify. Selection and use of such tools and commer- 
cial NIDS (in particular) should bear this in mind 
along with their flexibility to insert the “‘latest”’ trigger 
information that may be provided by security teams 
and organizations. 


The above shows the request for packet statistics and 
the flood results. 
## ngrep -i -x “switch tijgu" udp 
Ue 1O2. 1687-015 4001) 7-192 1682 hO. 2 8753 
Tol 7.692 FG - 63) 68.20" 74 switch t 
69 sbal (67 275 2 Oo Owe ijgu 204 
RO 33 20> 30: 20: 32. 30 O25 2.90 Related work 
U 92991682102 Dey 5 22° =>- 192. 168.10). Ww0433 ; 
73°77 69 74 63 68 65 64 Baya We present a brief overview, in chronological 
NO Tis 6064-67) 15 200. 35 ied order to the best of our knowledge, of DDoS tools that 
20 32 39 36 196 have been mentioned publically. 


pti Early tool 
This previous example shows the directive from TERE eons 


We handler enntch tothie handler The early tools appeared in early summer 1998. 


They were clumsy attempts to naturally evolve beyond 


For specific signature detection, one could also coordinated attacks [17], but nevertheless laid the 
use Snort [25]. See the caveats below. foundation to the subsequent tools. The first of them, 
Active Scanning fapi, featured UDP, TCP (SYN and ACK), and ICMP 

Scanning the network for open port 20432 will Echo floods. Its handler to agent communication was 
reveal the presence of a handler on your local area net- UDP-based. It did not provide easy controls for setting 
work. up the DDoS network, and did not handle networks 


over 10 hosts very well. The second one, fuck_them, 
rt te ai W ue eee was a distributed ICMP Echo Reply flooder, where the 
Die re rs ed ya oes at aan eee attacker either supplied the source address to spoof or 


oes a aes Messages ee ce oe ae randomized source addresses were generated (all 32 
word to all nodes on a network on the default bits of the IP address). 


port 18753 will generate traffic back to the detector, 
making the agent believe the detector 1s a handler. 


For detecting idle agents, one could wnite a pro- 


Trinoo and variants 


Trinoo surfaced in the early summer 1999. It has 
been extensively scrutinized, and we refer to [11] for a 
thorough analysis. The tool is capable of only generat- 
ing UDP packet floods without source address forgery, 


There are also two excellent scanners for detect- 
ing DDoS agents on the network: Dittrich’s “dds” 
[15] and Brumley’s “‘rid”’ [6]. 


“dds” was written to provide a more portable but has full control features. It was capable of crip- 
and less dependent means of scanning for various pling the network of the University of Minnesota [10, 
DDoS tools. (Many people encountered problems with 11] for three days, leading to a workshop on the sub- 
Perl and the Net::RawIP library [24] on their systems, ject [7]. Trinoo has mutated at least twice over the last 
which prevented them from using the scripts provided year. 
in [11, 12, 13].) Due to time contraints during coding, TEN And Variants 


“dds” does not have the flexibility necessary to spec- 
ify arbitrary protocols, ports, and payloads. One would 
need to modify the source slightly to detect shaft 
agents or handlers. A modified version of “‘dds”’, 
geared towards detecting only “Shaft” agents, is 
available [9, 16]. 


TFN, a.k.a. Tribe Flood Network, was intro- 
duced in late summer 1999. With its limited control 
features, it still provided UDP packet flood attacks (it 
gave homage to Trinoo by calling it “trinoo emula- 
tion”), TCP SYN flood attacks, ICMP Echo flood 
attacks, and Smurf attacks in a distributed fashion. It is 


A better means of detecting shaft pander and capable of spoofing either all 32 bits of the IP source 
agents would be to use a program like “rid”, which address, or just the last 8 bits. As with Trinoo, this tool 
uses a more flexible configuration file mechanism to has been analyzed thoroughly [12]. 


define ports, protocols, and payloads. TFN2K, or TFN2000, is a further development 


A sample configuration for “rid” to detect the effort on the basis of TFN. It provides the same 
Shaft control traffic as described: attacks as TFN, but can randomly do them all at once. 
Start ‘Share Encryption of the control traffic was added to improve 
send udp dport=18753 the security of the DDoS network and evade signature 
datartalive cijgu hia S984" detection. Control traffic uses a superposition of UDP, 
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TCP and ICMP, using oblivious transfers, 1.e., the 
receipt is not acknowledged. For a brief review, see 
[2]. 

Stacheldraht and Variants 


Stacheldraht, German for ‘“‘barbed wire,” appar- 
ently evolved out of Trinoo and TFN. Analyzed in 
[13], it has full control features, the same basic attacks 
and source address forgery as TFN, and as a twist, a 
Blowfish-encrypted control channel for the attacker. 
Mutated into StacheldrahtV4 in early 2000, it further 
mutated into Stacheldraht v1.666, which adds TCP 
ACK and TCP NUL packet flood attacks, and precon- 
figured Smurf attacks. 


Mstream 


As the name suggests, Mstream is a ‘multiple 
stream”’ tool, in reference to the very efficient point- 
to-point stream TCP ACK flooding tool. It has very 
limited control features and randomizes all 32 bits of 
the source IP address. For a review of this tool that 
appeared in the spring of 2000, please see [14]. 
Omega 

Omega, which appeared in early summer 2000, 
features TCP ACK packet flooding, UDP packet 
flooding, ICMP flooding, IGMP packet flooding, and 
a mix of all four floods. Similar to Shaft, it provides 
statistics on the floods it produces. It randomizes all 
32 bits of the source IP address, and introduces a chat 
function for communication between attackers. 


Trinity and Derivatives 


Trinity, and its closely related mutation Entitee, 
take a new approach on the DDoS model. Rather than 
relying on a handler network, it takes advantage of an 
existing Intemet Relay Chat (IRC) network for its 
handler-to-agent communications, making a channel 
on IRC the “handler.”” Besides the up to now well- 
known UDP, TCP SYN, TCP ACK, TCP NUL packet 
floods, it introduces TCP fragment floods, TCP RST 
packet floods, TCP random flag packet floods, and 
TCP established floods, while randomizing all 32 bits 
of the source IP address. 


myServer 


In contrast to the sophistication of Trinity, yet 
released around the same time in summer 2000, 
myServer is a simplistic DDoS tool. It relies on exter- 
nal programs to provide the denial of service. 


Plague 


As a third tool in the same generation as Trinity 
and myServer, it has become obvious Plague was 
designed by attackers who are reading these reviews 
and incorporating new improvements based on them. 
This tool provides TCP ACK and TCP SYN flooding, 
with what are claimed as fixes over previous TCP 
ACK flooding tools. 


Defenses and Countermeasures 


There is no simple solution that would offer one 
hundred percent protection against these types of 
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tools. There are, however, a number of steps that can 
be taken to minimize the impact [16]. Several pro- 
posed schemes are emerging for adding traceability to 
TCP/IP packets. 


What is necessary to defend? One needs to 
defend the hosts, the local net, and the backbone 
infrastructure. As per the recommendations in [7], cer- 
tain ingress and egress filtering can minimize the 
impact of denial of service attacks that use spoofed IP 
source addresses by eliminating illegitimate IP source 
or destination addresses. In practicality, this will not 
reduce the impact of DDoS tools that do not spoof 
their address or only spoof the last 8 bits of the IP 
address, making it appear to be originating from the 
local network that the agent resides on. 


One also tries to track floods and identify their 
source in order to shut them down one way or another, 
Or minimize the impact. In general, identifying the 
source of a spoofed IP address requires the collusion 
of the intermediate hosts in the path between the 
actual source and the victim suffering from the denial 
of service attack. Bellovin’s ICMP traceback message 
scheme [4] addresses that problem by forwarding a 
signed copy of the transient packet traversing the 
router in a probabilistic manner. In the proposed ver- 
sion, one in every 20000 packets triggers such behav- 
ior, In order to avoid an additional denial of service. 
Savage et al. take a different approach [26] in inserting 
partial network path markings into the packet travers- 
ing the router in a probabilistic fashion, rather than 
creating an entirely new packet. Their Fragment 
Marking Scheme, as pointed out by Song [28], lacks 
the scalability of dealing with large DDoS networks, 
causing a large number of false positives. Song’s 
marking schemes provide more efficient traceback 
under large scale (say 1500 agents) attacks. Stone sug- 
gests an IP overlay network to achieve the tracking 
and forwarding of interesting packets in his Center- 
Track [30] scheme. 


As mentioned above, anomaly detection can pin- 
point the presence of a flood in the first place. Signa- 
ture detection can either locate known control traffic 
(either attacker to handler, or handler to agent) or 
responses from victims. 


Future Trends and Evolution 


Ever since the introduction of Trinoo, at least 
eight new tools with varying degrees of sophistication 
and aimed at creating distributed denial of service 
have been discovered in a time span less than one cal- 
endar year. It is difficult to predict the trends of these 
tools without ending up being the trendsetter or spark- 
ing a new idea for the attacker. These tools have 
destructive potential and one should remain cautious 
as to the future directions. It is safe to assume, how- 
ever, that the trends described in the section on related 
work will continue, namely in creating distributed 
variants of existing point-to-point tools. 
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(DDoS) attack tools. He can be reached at 
dittrich@cac.washington.edu. 


Conclusion 


“Shaft” 1s another DDoS variant with indepen- 
dent origins. The code recovered did appear to be still 
in development. Several key features indicate evolu- 
tionary trends as the genre develops. Of significance is 
the priority placed on packet generation statistics 
which would allow host selection to be refined. The 
analysis of the code and binary was greatly enhanced 
by the capture of attack preparation and command 
packets. The captured packets made it possible to 
assess the impact of a single agent that managed to 
saturate the network pipe. 


The version analyzed had hooks which would 
allow for dynamic changes to the master host and con- 
trol port but not the agent control port. However such 
items are trivially incorporated and must not be taken 
to be indicative of any current versions which may be 
in active use. The obfuscation of master IP, ports and 
passwords used a relatively simple form of encryption 
but this could easily be strengthened. Evolutionary 
findings confirm that information flows back to the 
authors and cause incorporation of counter-counter- 
measures as the spiral continues. 


The detection of DDoS installations will become 
very much more difficult as such metamorphosis tech- 
niques progress, the presence of such agents will still 
be more readily determined by analysis of traffic 
anomalies with a consequent pressure on time and 
resources for site administrators and security teams. 
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Appendix 1: Agent Commands 


Accepted by agent and replies generated back to 

the handler: 

size <size> Size of the flood packets. Generates a 
“*size”’ reply. 

type <0|1|2|3> Type of DoS to run 0 UDP, 1 TCP, 2 
UDP/TCP/ICMP, 3 ICMP. Generates a “‘type”’ 
reply. 

time </ength> Length of DoS in seconds. Generates 
a ‘“‘time’”’ reply. 

own <victim> Add victim to list of hosts to perform 
denial of service on. Generates a “owning” 
reply. 

end <victim> Removes victim from list of hosts 
(see ‘““own”’ above). Generates a ‘‘done”’ reply. 
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stat Requests packet statistics from agent. Gener- 
ates a “pktstat”’ reply. 

alive Are you alive? Generates a ‘alive blah” 
reply. 

switch <handler> <port> Switch the agent to a 
new handler and handler port. Generates a 
“switching” reply. 

pktres <host> Request packet results for that host 
at the end of the flood. Generates a “pktres” 
reply. 

Sent by agent: 

new <password> Registering with the handler 

pktres <password> <sock> <ticket> <packets 
sent> Packets sents to the host identified by 
<ticket> number. 


Appendix 2: Handler commands 


This is an overview of the command structure: 

mdos <host list> Start a distributed denial of ser- 
vice attack (mdos = massive denial of service?) 
directed at <host list>. Sends out “‘own host” 
and ‘‘pktres’’ messages to all agents. 

edos <host list> End the above attack on <host 
list>. Sends out ‘“‘end host” messages to all 
agents. 

time </ength> Set the duration of the attack. Sends 
out ‘‘time </ength>” to all agents. 

size <packetsize> Set the packetsize for the attack 
(8K maximum as seen in source). Sends out 
“*s1ze <packetsize>”’ to all agents. 

type <UDP|TCP|\ICMP|ALL> Set the type of 
attack, UDP packet flooding, TCP SYN packet 
flooding, ICMP packet flooding, or all three. 
Sends “‘type <type>”’ to all agents. 

+node <host list> Add new agents. 

-node <host list> Remove agents from pool. 

ns <host list> Perform a DNS lookup on <host 
list>. 

Inod List all agents. 

Itic List all pending tickets (transactions). 

pkstat Show total packet statistics for agents. 
Sends out “‘stat”’ request to all agents. 

alive Send an “alive” to all agents. A _ possible 
argument to alive is “‘hi”’ 

stat show status values (length, type of DoS, packet 
size). 

switch become the handler for agents. Send 
“switch” to all agents. 

ver show version. 

whoami returns “God ”’. 

exit self-explanatory. 
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ABSTRACT 


This paper presents YASSP, Yet Another Secure Solaris Package, a set of tools developed to 
help a systems administrator to secure a Solaris host. It explains the internal mechanism used by 
YASSP to implement these security changes so that a systems administrator can either use this tool 
in its current form and localize it, or modify the package source to match his needs. 


YASSP is composed of the SECclean package and a set of optional packages providing 
common useful security related tools. The SECclean internal mechanism used to modify the 
existing operating system is implemented through the Solarns package mechanism and provides a 
full un-installation procedure. It has required the development of a specific library of shell 


functions, to ease file manipulations. 


The YASSP project provides to the community an easy path to secure a Solaris host. It has 
taught us a lot about Solaris internals and package manipulation. YASSP 1s still young, and ready 


to be enhanced. 


Introduction 


The project YASSP was started back in 1997 to 
fulfill the need for installing exposed Solaris servers. 
For such servers, Sun’s Solaris default installation has 
the following weakness: 

e As standardly installed, Solaris is running many 
network services. This violates the principle 
that a machine should provide only those ser- 
vices that its owner intends to provide. 

As standardly installed, all network services are 
available to any host that requests them, violat- 
ing the principle of "offer services only to those 
you intend to use them" 

As standardly installed, system logging is 
inconsistent. 

As standardly installed, the system does not 
issue security-warning banner messages to peo- 
ple who attempt to use It. 


YASSP’s current version was created at Xerox 
Palo Alto Research Center, as a configurable tool to 
secure a Solaris host. A more complete history of the 
project can be found in later in this paper. 


YASSP can be used on Solaris 2.6, 7 or 8, sparc 
or Intel architecture to enhance the security of an end- 
user workstation or a server. 


Philosophy 


Before looking at YASSP internals, let us under- 
stand the concepts used to build YASSP on. These 
concepts had driven YASSP implementation, had cre- 
ated their own problems and provided solutions, but 
have stayed constant over the existence of YASSP 
which has made it very stable. 

e It must play by Sun’s rules and not fight Sun: 
we follow as close as we can Sun standard and 
rules. 
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e YASSP tries to be as closely integrated to 


Solaris as it can. It was built and distributed as 
a Solaris package. YASSP will follow package 
rules and Sun standards when they exist. When 
standard does not exist, YASSP will try to 
accommodate the different choices the systems 
administrator may have made. 

It can be installed and un-installed without 
destruction and restores the same context as 
before. Particular attention was given to these 
points, to be sure that the systems administrator 
will never lose some files or modifications 
done after YASSP installation to any YASSP 
configuration files. The goal here is double: be 
sure that any changes done by YASSP can be 
undone, but also be sure that any changes done 
after will not be lost in case of un-installation. 
That is true for all components of YASSP. 

It should run on a minimal installation, like the 
core choice of the Solaris installation, or any 
other. 

This is one of the most important, and the most 
limiting rules, as it directly drove choices we 
made in the tools and language used in YASSP 
implementation. Except for a few binaries, the 
core of YASSP is written in Bourne shell, uses 
sed and [n]awk. 

Additional scripts, not part of YASSP but 
referred to as examples on the web documenta- 
tion may use tools not included in the core 
install, like Perl. 

Secure and closed by default... 

... but can be opened up after the installation or 
modified to be localized. 

YASSP default behavior is to be secure. For 
each configuration choice YASSP offers, it will 
by default install the one the YASSP team of 
developers had consider the most secure. 
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e Most of these configuration choices are easily 
reviewed and modified after YASSP install. 

e Using a three-layer implementation: from the 
security issue we are trying to solve, we gener- 
ate a list of modifications that need to happen; 
this list is implemented by the package installa- 
tion (File replaced, file edited, file created...) 


Being burlt as a Sun package, and following Sun 
rules, this tool cares about the Solaris package 
database, corrects it when it 1s wrong, and keeps it 
accurate over time. 


Result of YASSP Installation 


After installing SECclean, most of your network 
services will be tumed off and most of the processes 
started by default under Solaris will not run anymore. 
Your server will be hardened, perhaps too much. That 
is the SECclean default: if you do not modify its con- 
figuration, or if its configuration does not exist, it will 
be closed and secure. That fits pretty well for 
installing an exposed server, but not an end user work- 
station. 


SECclean modification to your system can be 
summarized as: 

e Deleting unwanted files (/etc/auto_ home, 
/etc/auto_ master, /etc/dfs/dfstab, 
/var/spool/cron/crontabs/adm, 
/var/spool/cron/crontabs/Ip, and 
/var/spool/cron/crontabs/uucp). 

e Controlling most of the startup scripts through 
/etc/yassp.conf. 

e Changing default environment setting to make 
your system more secure (umask, password 
length, PATH) and make some of these vari- 
ables available through /etc/yassp.conf (/etc/ 
profile, /etc/default/login, /etc/de fault/su, /etc/ 
default/passwd, /etc/skel/local.cshrc, /etc/skel/ 
local.profile, /etc/.login, /etc/rmmount.conf) 
and setting default umask for startup script 
(/etc/init.d/umask.sh). 

Enabling banner files (/etc/motd, /etc/issue, 
/etc/ftp-banner). 

Controlling access to the system and to specific 
commands (/etc/pam.conf, /etc/default/login, 
/.rhosts, /etc/hosts.equiv, /etc/cron.d/at.allow, 
/etc/cron.d/cron.allow, /etc/default/sys-suspend, 
/etc/ftpusers, /usr/dt/config/Xaccess, and /etc/ 
dt/config/Xaccess). 

Simplifying network startup, but keeping back- 
ward compatibility through /etc/yassp.conf 
(/etc/init.d/inetsvc, /etc/init.d/inetinit, and /etc/ 
init.d/netwo rk). 

More control on RPC services (keysery and 
rpcbind with TCP wrapper) [15] (/etc/ 
init.d/rpc). 

Tuming off available network services (/etc/ 
inet/inetd.conf). 

System parameters tuning, attempting to pre- 
vent and log stack-smashing attacks (/etc/sys- 
tem). 
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e Network stack performances and security tun- 
ing [16] (/etc/init.d/nettune and /etc/de fault/ 


inetinit). 

e Adds few common _ services definitions 
(/etc/inet/services). 

e Establishes standard syslog configuration 
(/etc/syslog.conf). 


e Miscellaneous and logging (/etc/shells, /var/ 
adm/loginlog, /etc/norouter, and /etc/inittab). 

e Package database correction and filemode 
changes to make them more secure (clean-up 
and fix-modes). 

A more detailed and up-to-date explanation of what is 
actually done can be found on the web, at: 
http://www. yassp.org/internal.html . 


On top of SECclean, YASSP will provide 
integrity check (tripwire) [4], TCP and RPC controls 
(tcpd and rpcbind with tcp wrappers) [15], encrypted 
channel (ssh) and others log rotation and version con- 
trol of system files (PARCdaily, GNUrcs, and 
GNUgzip). 

An example of how a system looks after the 
reboot can be found on Appendix 1. 


First, as Sun packages have an important place in 
YASSP implementation, let us explain what they are 
and how they work. 


Sun Packages Overview 


In the Solaris world, application software or sys- 
tem components are delivered in units called 
packages'. A package is a collection of files and 
directories required for implementing a new function, 
and is usually designed and built by the developer of 
this new function (new applications, system enhance- 
ment) after completing the development of the code. 


The components of a package fall into two cate- 
gories: package objects which are the application files 
to be installed, and control files which control how, 
where, and if the package 1s installed. 


Control Files 


The control files are divided into two categories: 
information files, and installation scripts. Some of 
these control files are required and some are optional, 
installation scripts are always optional. The installa- 
tion script must be composed of Bourne shell com- 
mands. 


Installation script performs customized actions 
during the installation of your package, especially pro- 
cedure scripts defining actions that occur at particular 
points during package installation and removal, and 
the class action script, which defines a set of actions to 
be performed on a group of objects. Four procedure 
scripts are predefined: preinstall, postinstall, prere- 
move, and postremove. 


'This part does not intend to be a package tutorial, but 
rather a short introduction to provide the reader enough 
knowledge about them to understand YASSP intermals [13]. 
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Package Objects 


These are the components that make up the 
application. They can be: files (executable or data), 
directories, named pipes, links or devices. 


Package objects are described in the prototype 
files, one line per single deliverable object. An object 
description includes its location (pathname, relative or 
absolute), attributes (owner, group and permission 
mod), class and file type. 


Object classes allow a series of actions to be per- 
formed on a group of package objects on installation 
or removal. The sed’s predefined class provides a 
method for using sed instructions to edit files upon 
package installation and removal. See Sun’s documen- 
tation for more details about object classes. 


Package Delivery, Installation, Registration and 
Checking 


At the installation of a package, all of its objects 
are registered in /var/sadm/install/contents. Informa- 
tion stored in this file is either static information found 
in the package’s information files (package name, file 
type of the object, attributes, class), or dynamic infor- 
mation generated at the installation by the pkgadd 
command, like the date, the size and the checksum 
(sum(1)) of the object, which provides a good integrity 
check but a poor authenticity check. 


The accuracy and the integrity of the various objects 
of an installed package can be verified using the 
pkgchk(1) command. The ‘-n’ option, which tells 
pkgchk not to check volatile or editable files, should 
be used in a post-installation check. Checks are done 
on the existence of each object, their recorded 
attributes, their size, and checksum. 


Package information can be updated manually 
anytime using the instal/f(1M) and removef(1M) com- 
mand. 


Packages are the recommended way to deliver 
Solaris applications. Solaris packages provide tools 
and methods to install new objects, and modify or 
delete existing ones through either the action class or 
the installation script. They also provide a basic mech- 
anism to save files modified and will record all 
installed files, including their attributes and a basic 
checksum. 


Packages are not perfect but exist on every 
Solaris system. 


SECclean Use of the Package Mechanism 


SECclean is the core of YASSP, implementing 
most of the security tuning. SECclean is built as a 
package, using the rules we explained in the last sec- 
tion. To implement this security tuning, SECclean will 
modify the existing Solaris system by adding, delet- 
ing, replacing or modifying various files. 


Files can be sorted into different groups: created, 
deleted, replaced, and modified files and startup 
scripts. 
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New files, deleted files and replaced files are 
three categories of files easily understood. Sometimes, 
it is more prudent to modify an existing file instead of 
replacing it with our version, as they may be different 
on each system. 


Startup files, or Run Control Scripts as Sun doc- 
umentation named them, are Bourne shell scripts to 
control run level changes. Each run level has an asso- 
ciated rc script located in the /sbin directory: rc[0-6S]. 


Run control scripts are located in the /etc/init.d 
directory with links from the corresponding run con- 
trol scripts in the /etc/rc[0-6S].d directories. Due to 
their specific functions, and requirements, startup files 
are handled as special cases by SECclean. 


Since nothing can be simple, exceptions will 
exist to this classification. SECclean is compatible 
with three versions of Solaris, and files needed to be 
replaced may be dependent of the version. Other files 
we want to modify will need special care, as these 
modifications can be more complex. Binary files 
installed by SECclean need to be handled depending 
of the architecture of the target. Last, some files may 
need to be modified only if they exist. 


File Management Under SECclean 


As explained in the Philosophy section, SEC- 
clean must be un-installable, and the state of your sys- 
tem after a SECclean installation and un-installation 
should be the same as before. All the information 
associated with a file (its contents, its attributes 
(owner, group and permission mode) and also the 
package information related to this file) must be 
extracted and saved at SECclean installation, so that 
the removal step will be able to restore it. These 
backup and restoration procedures have to be created, 
as packages provide you a guideline on how it should 
be done, but it is up to the package developer to 
implement such features. A Bourne shell? library 
named cleanlib.sh provides functions to help deal with 
package registration, files backup and restoration. To 
perform any file operations, SECclean uses cleanlib.sh 
functions, as package operations provided by Solaris 
are very primitive. 


Cleanlib provides the following functions: 

e Install _fileQ: It installs a file by moving it from 
its prefixed name to its real name. It keeps a 
copy of the installed file so that any modifica- 
tion to it can be tracked, and registers the file as 
part of the SECclean package. 

e Install RC_fileQ: Same function as 
Install file(Q) but to install a startup script. It 
also creates the symbolic links needed under 
the /etc/rc[0-6S].d directories. 

e Backup user_file(): Copies a file and re-create 
its path under a backup directory. 


2Why Boume shell? These functions will be used in the 
package installation scripts, which must be written in 
Boume shell, and we should be able to use this library on a 
core installation. 
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e Save_and move _file(): Moves the indicated 
file and its package information into a backup 
area and then un-registers it from the package 
database. 

Disable RC(): Removes links to a startup 
script. It will backup all package information 
found. 

Disable_Init(): Searches for any registered link 
to a startup script and calls Disable RC() for 
each of them. 

RCconfized_Init(): Modifies a startup script to 
control its start through /etc/yassp.conf. Will 
also update /etc/yassp.conf for each startup 
script modified. 

Create RCconf(): Initializes the static header 
of /etc/yassp.conf. 

DE_RCconfized_Init(): Undoes the modifica- 
tion done by RCconfized_Init() on startup 
scripts. 

Restore RC(): Restores a startup script links 
from the saved information and updates the 
package database. 

Restore_file(): Restores a file from its backup 
and updates the package database. 

Init. preremove(): Initializes the value for the 
backup directory for files modified since 
YASSP installation. 

Cmp_and_backup file(): Checks to see if any 
previously installed files have been modified 
and backs them up if so. 


To delete an existing file, we first save its current 
package information, move it to the package save 
directory and un-register it from its original packages. 
This is done by the Save_and_ move _file function. 


Replacing a file involves more steps. We must 
first deliver the new file under a different name, in 
order to not overwrite the existing one. The conven- 
tion for SECclean is to install the new file with a name 
prefixed by "SECclean_’. Then, we must delete, as 
explained in the last section, the existing file, move 
the new file to its real name, un-register the prefixed 
name from the SECclean package and register the real 
name init. Also, as SECclean wants to be able to track 
any changes to these files after its installation, a copy 
of the original installed file is saved. These operations 
are done using the Save_and_ move file and the 
Install file functions. 


Files that need to be modified are defined as part 
of the sed class in SECclean prototype. No specific 
action 1s required as the associated sed script handles 
the installation and un-installation phase. 


For startup scripts, we wanted to offer the sys- 
tems administrator the choice of running the different 
startup script present, and we wanted this choice to be 
as convenient as possible. The solution chosen in 
SECclean was to modify the existing script so that it 
will start only if a shell variable is set to YES in 
YASSP’s configuration file /etc/yassp.conf. The name 
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of this shell variable is based on the name of the 
script, stripped of all non-alphabetic characters and 
capitalized. The function RCconfized_ Init is used to 
modify each original startup script. 


The Disable_Init function is used to backup and 
delete any information associated with a startup script. 
It will, from the name of the startup script under 
/etcAnit.d, find all the existing links to it from any 


“/etc/rc[0-6S].d/ directory, record all these links and the 


package information associated with them, save the 
content of the original startup scrip and delete them 
all. It 1s used when we need to replace or delete a 
startup script. 


Cleanlib also provides the reverse functions, 
used at the removal phase of the package to restore the 
state and the contents of these files. 


Exceptions exist: some startup scripts need fur- 
ther modification, some startup scripts need to be 
replaced by our version but depending on the operat- 
ing system version, some files need to be installed 
only if they were not already present. 


All these functions keep track of the list of pack- 
ages they touch, so that when we are done, we know 
which changes have to be validated. 


Other Tasks Performed by SECclean 


In addition to the file operations we described, 
SECclean also needs to do the following tasks: 


User Convenient Backup 


We found it very useful and convenient to copy 
the tree of files SECclean will touch and provide it as 
an informational copy to the systems administrator. 
This copy is not used by the package backup mecha- 
nism, but provides the systems administrator an easy 
way to see which and how files were changed. The 
backup is done by default under /yassp.bk (See the 
Putting it all together’ section for more details). 


Password File Cleanup 


The password file will be checked and cleaned 
up if needed, to lock all non relevant system accounts, 
change the shell of all locked accounts to a binary 
which will syslog any use of these accounts before 
exiting. Password file cleanup is implemented in the 
clean_passwd script, part of SECclean, and can be re- 
run anytime after SECclean installation. 


Operating System Cleanup 


When you install Solaris, even if Solaris compo- 
nents are built as packages, you will find some inco- 
herence if you check the package integrity. These 
incoherencies reflect either files modified during the 
install, or files registered incorrectly, or both. Using a 
shell script, which is very OS dependent, SECclean 
will correct these incoherencies and leave the package 
database clean. 


After, using Casper Dik fix-modes program for 
Solaris [3], SECclean secures various files permission 
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and ownership. It does this by removing group and 
world write permissions of all files/devices/directories 
listed in /var/sadm/install/contents, with the exception 
of those pre-listed in fix-modes program. 


Putting It All Together 


Installation Process 


SECclean makes use of the preinstall and postin- 
stall procedure scripts. 


Preinstall 


The preinstall script first determines under which 
directory the SECclean component will be installed. 
Some components of YASSP, which may be installed 
later, have pathnames hard coded in their binary’. On 
the other hand, systems administrators may want to 
install their software in another place. If /opt/local 
exists and if we can change dir to it’, SECclean uses 
/opt/local as its default installation path. If we cannot 
change dir to /opt/local, but if /usr/local exists and we 
can change dir to it, /opt/local is created as a symbolic 
link pointing to /usr/local. If none of them exists, 
/opt/local is created as a directory. 


Then, the preinstall script initializes the two shell 
variables $SECBCK and $CLEANUPDIR, if they are 
not already defined. $SECBCK is used as the path- 
name under which the user convenient backup will be 
done. Its default value is /yassp.bk. $CLEANUPDIR 
is the path where the scripts and binaries related to the 
operating system clean-up will be installed. Its default 
value is /var/sadm/clean-up. 


Next, the preinstall script defines four shell vari- 
ables, corresponding to categories of files. The vari- 
able $RCCONF corresponds to an exhaustive list of 
startup scripts. Startup scripts are always referred to 
by their name under the /etc/init.d directory. This list 
is exhaustive, gathering all the startup script names we 
want to control in all Solaris versions we support. 
Next variable is $NRC, defining the startup script we 
want to replace with our own OS dependent version. 
Three startup scripts are part of this list: inetsvc, ine- 
tinit and networks (only on Solaris 8). They are the 
scripts driving the network setup at the boot time. The 
last two variables defined are the list of files to be 
deleted ($SD) and the list of files to be replaced 
($SA). All these variables are recorded in a file 
($PKGSAVE/.PROC Init Var), so that the other 
scripts used in SECclean installation or removal can 
share the same lists. For an up-to-date list of the files 
defined in each of these variables, please consult 
http:://www. yassp. org. 


The user convenient backup 1s the last thing done 
by the preinstall: all files SECclean intended to touch 
will be copied under $SECBCK/Before ‘date 
+% Y.Yom.%d-%H.%M.%S° . 


J3An example is the RCS package, compiled with the 
GNuUdiff: RCS binary has the pathname of GNUdiff hard 
coded. The default pathname we used is /opt/local 

4/opvlocal can be a symbolic link to any existing‘directory. 
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Installation 


It is at the installation phase that all the package 
objects defined in SECclean will be installed. Objects 
which correspond to files we want to replace have 
been registered and will be installed with a name pre- 
fixed by ’SECclean_". For objects dependent on the 
operating system version (inetsvc for example) or on 
the architecture of the host (binary), all possible 
instances were registered and will be installed using 
the OS version or the architecture as a suffix. 


All sed scripts associated with any package 
objects part of the sed class will be then executed, and 
the targeted file modified. 


Postinstall 
The postinstall script is the most complex one. 


After reading the PROC_Init_Var file created by 
the preinstall to learn the value of the different cate- 
gories of the file, it first disables startup files we will 
replace later, by calling the Disable Init function. 


Next, all the files listed in RCCONF are modi- 
fied, using the RCconfized_Init function. For each of 
them, a sed script is generated and applied to them. 
From the result of which startup files were present on 
this system, and from some static information, the 
yassp.conf file is generated. 


All the files we will delete or replace are backed 
up and moved out using Save _and_ move file. At this 
step, SECclean is done modifying existing packages, 
and it can now close and validate all the removals 
done to the package database. 


The next steps deal with SECclean package 
objects. For files that are dependent on the operating 
system version or on the machine architecture, the 
right version is kept and others are deleted and unreg- 
istered. SECclean package information is updated and 
the database is closed. All the files we want to replace, 
that have been installed using ’SECclean_’ as a prefix 
in their name, are renamed to their right name. The 
new inetinit, inetsvc and networks? startup scripts are 
installed and symbolic links are created from the 
appropriate /etc/rc[012].d directories, and _ special 
cases are handled for specific files. SECclean database 
is then closed again, to validate all these changes. 


We are now done with file installation and pack- 
age database update. Tuning to some newly installed 
files like the /etc/system is done depending on the 
operating system version, syslog files used by the 
newly installed syslog.conf file are touched, and the 
password file is cleaned up by calling clean_passwd. 


The last thing done is the operating system clean 
up, which includes the package database correction 
and running fix-modes. 


We are done: SECclean is installed. 


‘These startup scripts are treated as a special case: we want 
them to execute the strict minimum instead of handling all 
the cases planned by Sun (DHCP for example). The new 
scripts provided are always based on the original ones. Set- 
ting variables in yassp.conf, can restore the original Sun be- 
havior of these scripts. 
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Removal 


The removal phase has two main tasks: restoring 
the state your workstation as it was before SECclean 
installation, but also keeping a copy of any files, 
installed by SECclean, and modified by the systems 
administrator so that these modifications will not be 
lost. It will use the preremove and postremove proce- 
dure scripts to implement these tasks. 


Preremove 


After reading the PROC _Init_Var file created by 
the preinstall to learn the value of the different cate- 
gories of files, 1t checks all files replaced by SECclean 
against the originals, and keeps a backup copy under 
/var/tmp/SECclean.Backup_$$ if they have been mod- 
ified. Next, it makes a copy of the cleanlib.sh library, 
as this file will be removed in the removal phase, and 
the postinstall needs it. 


Removal 


All files registered as part of SECclean prototype 
are deleted. Sed scripts associated to package object of 
the sed class will be executed to undo any modifica- 
tions done. 


Postremove 


After reading the PROC _Init_Var file created by 
the preinstall to learn the value of the different cate- 
gories of files, it will unregister from the package 
database all files registered as part of SECclean in the 
postinstall script, listed in the $SA and $NRC vari- 
ables, and close the database. 


Next step is to restore the original contents of 
files SECclean has deleted or replaced, and re-register 
them in the package database with their original 
attributes. 


The modifications to control the start of the 
startup scripts are undone, and the package database 1s 
closed to validate these changes. 


Last, the copy of the cleanlib.sh library is 
deleted. 


How We Tested It 


This complex process was validated by installing 
(and un-installing) YASSP on various hosts, for test- 
ing as well as being part of the day-to-day work. 


The final state after the installation is tested 
using tools like titan [8], ISS ... and of course we 
asked the beta testers to use their expertise. 


The SANS team is also very helpful for advice or 
consensus on proposed changes, or for pointing out 
existing documentation and tools [2, 5, 6, 7, 8, 9, 10, 
11, 12, 13, 14]. 


Particular attention was given to test the package 
removal function. Testing can be time consuming, 
especially when the package is broken and corrupts 
the package database. Use of Vmware virtual machine 
running Solaris for testing 1s a big time saver. 
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YASSP Other Goodies 


YASSP also offers to install a set of useful and 
common tools, built as separate packages. It includes: 

e GNUrcs: RCS 5.7 and diff 2.7 [GNU] to ease 
version control of your files. 

e GNUgzip: gzip 1.2.4a [GNU]: file compres- 
sion, 

e PARCdaily: Some daily scripts, logs rotation, 
backup and RCS for system files... Need 
GNUgzip and GNUrcs. It is mainly an example 
to show people what they can do. 

e WVtcpd: tcp_wrappers 7.6 + rpcbind 2.1 
[Wietse Venema]: tcp wrapper, and Wietse’s 
version of rpcbind compiled with tcp wrapper 
library, for people who need to run RPC. 

e PRFtripw: Tripwire 1.2 [Purdue Research 
Foundation of Purdue University]: Integrity 
checking tools 

e SSHsdi: version 1.2.30-SDI, build with 
SecurID support (See ftp://ftp.parc.xerox.com: 
/pub/jean/sshsdi/README ). 

All these packages will, when installed, take all the 
necessary steps to be ready for use. For packages rely- 
ing on configuration files, some proposed configura- 
tion files are provided, and they will be installed as the 
default configuration file if they were not already pre- 
sent. 


YASSP Installation 


What Can Go Wrong During the Install 


From our experience, nothing should go wrong if 
it is a fresh Solaris install. For an installation done on 
an existing system, you may find some conflicts dur- 
ing the package installation with files or directories. 
Example of Installation 

+ uncompress yassp.tar.Z 
- Car xve yassp.tar 
# cd yassp 
# ./install.sh 
. check and modify all 
the configuration files .. 
# reboot 


A commented installation log can be found in 
Appendix 2. 


What May Be Broken After? 


SECclean and YASSP defaults are to disable (or 
to ’secure’) most of the existing services. If you are 
planning to install YASSP or SECclean on an existing 
server, you must know what application(s) are running 
on it, and what network services and system resources 
are being used. Your work after the YASSP install will 
be to re-enable these services, and only these services, 
to have a system where you know why any process or 
service is running. YASSP will help you by providing 
some tools to monitor these changes and re-enable 
some security features after its installation, but YASSP 
cannot understand your setup as you can do. 
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Also, 1f your system had non-standard applica- 
tions or configurations, SECclean may have missed 
some tuning specific to your application. 


What To Do After? 


Most of YASSP configuration 1s handled by a 
single text file /etc/yassp.conf. A man_ page, 
yassp.conf{4) http://yassp/man/yassp.conf4.html, descnbes 
the different tuning that can be done through this file. 
A more generic description of what to do after the 
installation, not focused on SECclean, is maintained 
at: http://yassp/after.html . 


And do not forget to send us feedback about 
YASSP; we are eager to receive comments, sugges- 
tions or just acknowledgement that people are using It. 


Where Are We Now? 


As of October 10, 2000, beta#12 of YASSP is 
released on http://www.yassp.org . It has been tested 
and supports Solaris 2.6, 2.7 and 8, under sparc archi- 
tecture. Beta#l2 is used widely®, and is the release 
candidate for v1.0 [17]. 


YASSP is built on top of two independent com- 
ponents: 

e The core package, named SECclean, imple- 
menting the OS security modification, which 
also includes correcting the incoherence left 
after a standard Solaris install and modifies 
some filemodes to make the file hierarchy more 
secure. 

A set of additional packages, provided as an 
optional install as they represent some com- 
monly used features not implemented by 
Solaris. 

Systems administrators can use these tools choosing 
either the YASSP install.sh script included on the 
YASSP tar file, or to run each package or script manu- 
ally. 


The default install.sh script of YASSP will 
cleanup the Solaris install, modify the filemodes to 
secure various files and directories, install the core 
package and propose to install a set of additional pack- 
ages. 


After installing the core package, SECclean, the 
result is a system quite closed and secure, where most 
of the services have been tumed off, minimal pro- 
cesses are running, and with some security tuning 
applied at various levels of the operating system, from 
the network stack to the XDM configuration file. 


Future 


Still a lot to do, from gathering more feedback, 
to porting it to new Solaris releases, and improving the 
web documentation. 


6Widely is vague, but it is hard to be more precise. Since 
the YASSP site was announced, the logs show the number of 
downloads, but unfortunately, we have received very little 
feedback. 
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Two main directions can be guessed from the 
talk in YASSP mailing list: 

e Develop new tools to help monitor the system 
after the initial installation, to check that no 
modifications happen and be able to re-apply 
some security settings on-demand. 

« Develop a tool to manage some more restrictive 
file permissions, especially for SUID/SGID 
binaries, as an option. 


Conclusion 


The YASSP project has taught us a lot on Solaris 
internals, both from a security point of view and about 
the Solaris package mechanism. The complexity 
added by the way YASSP deals with files and package 
information has shown its strength as it provides a 
robust un-installation mechanism and eases a lot 
YASSP maintenance for any new version of Solaris. 


YASSP 1s used by various organizations, 1n vari- 
ous contexts from end user workstations to exposed 
servers. Feedback we received shows us that it 1s a 
useful tool that people appreciate. YASSP is young, 
and will continue to be improved, but we have to be 
careful to keep it focused on securing a Solaris host, 
and keep its size reasonable so that we can maintain it 


Last, we would like to thank all the people who 
have participated in the YASSP mailing list or have 
sent us feedback. Please continue. 
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History of the YASSP project 
A Long Time Ago... 


Back in 1997, we faced the task of having to 
deploy firewall and external servers in various loca- 
tions around the world. Even if the configuration was 
standard and well known, we realized quickly that, as 
the local Unix skills were unknown, we needed to help 
systems administrators and provide a way to install 
these servers so that we could be pretty comfortable 
that the result of this installation was what we 
expected to see. 


We started to write down some guidelines on 
how to install and then secure these servers. That was 
perfect for experienced systems administrators, but we 
quickly realized that something written does not mean 
that people will read it, and that this guideline was 
written for people understanding the context of the 
installation (skilled in network, Unix, Solaris), which 
was not always the case at these remote sites. 


We translated these guidelines to shell script, 
very primitive, which was supposed to do all the work 
automatically. It required good Unix skills from the 
systems administrator running it, was very touchy 
with the existing state of the server, which had to be a 
fresh installed OS, and was destructive in the sense 
that files were replaced without backup. There was no 
way to undo the damage caused by it, neither was 
there a way to use it to perform a different task than 
installing an external server as defined per our organi- 
zation. 


Xerox PARC 
In 1999, the author came to Xerox Palo Alto 
Research Center and was able to allocate more time to 


this project. The idea was to enhance the existing 
baseline and enable its use on end-user workstations. 
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The shell script was ported to the Sun package 
format, allowing it to be un-installed and integrated 
into our (PARC) default installation process. 


This package grew, including lots of suggestions 
from its users (mainly Xerox people) and new features 
or tuning gathered from various sources. 


At this time, it provided a way to harden a fresh 
Solaris installation to be used mainly on an exposed 
server (most of the services will be turned off), was 
un-installable if necessary. A public version was avail- 
able on our external ftp site and was used by various 
people. Basic documentation existed on the same ftp 
server. 


SANS 


With the SANS Institute, we created a team to 
work on tools for securing Solaris. This team (10 peo- 
ple at the beginning, about 160 now) was open to any- 
one who committed to spend time testing or reviewing 
existing tools and providing feedback. 


It drove some major changes on the existing 
tools. 


First, the scope was extended: it should still 
focus on exposed servers, but should be also suitable 
for end-user workstations. 


Also, it was modified to be permissive on the 
expected state of the operating system and able to be 
installed on existing install rather than requiring a 
fresh Solaris install. 


Some recommended security tools were added to 
it, implementing additional features not available in 
Solaris. 


At last, and on top of all the bug corrections and 
enhancements suggested by this team, the tool was 
modified to be more verbose, have better documenta- 
tion and provide the systems administrator an easier 
way to configure the desired security level. 


Of course, some problems were raised and some 
nice verbal argumentation happened, but life would be 
too quiet without that. 


Why Does It Still Exist? 


At the time it was started, there were not many 
tools available to secure Solaris, but now, various 
Other tools exist (Titan, Bastille-Solaris), and people 
might wonder why we spend our energy on creating 
and maintaining a separate tool rather than merging 
our efforts with existing tools. 


The first reason to keep YASSP alive was that it 
existed, was working, was used by people and was 
useful to them if we believe the feedback we received 
at the time. 


Also YASSP deals correctly with Solaris pack- 
ages, and will keep the database clean. This part was 
unique to YASSP, as well as the unique configuration 
file it offers. 


Last, we found that for some systems administra- 
tors, it was less confusing to have the minimum 
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interaction with the script, especially with less Unix 
experienced systems administrators. These systems 
administrators, with little or no Unix/Solaris experi- 
ence, may be required to install Solaris server in a hos- 
tile environment. YASSP will do that by default. 


Security 1s a field where having a choice of tools 
is a plus. More choices provide flexibility and more 
appropriate solutions. 
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Appendix 1: System with YASSP installed 


Rebooting with command: boot 
Boot device: disk File and args: 
SunOS Release 5.8 Version Generic 64-bit 
Copyright 1983-2000 Sun Microsystems, Inc. All rights reserved. 
configuring IPv4 interfaces: hmeO. 
Hostname: zeta 
Tweaking Solaris TCP/1P: Solaris 7 or -above (excellent) 
tweaking separate connection queues 
tweaking against SYN flood symptoms 
tweaking timeouts 
tweaking pMTU discovery interval and common timers 
tweaking misc. parameters 
applying security tweaks... 
tweaking windows, buffers and watermarks 
done. 
The system is coming up. Please wait. 
checking ufs filesystems 
/dev/rdsk/c0Ot0d0s5: is clean. 
petting. netmask. .of -nmeO to 255.255:.25 5:70 
syslog service starting. 
sshd starting. 
The system is ready. 
WARNING: To protect the system from unauthorized use and to ensure that 
the system is functioning properly, activities on this system are 
monitored and recorded and subject to audit. Use of this system is 
expressed consent to such monitoring and recording. Any unauthorized 
access or use of this Automated Information System is prohibited and 
could be subject to criminal and civil penalties. 


zeta console login: root 
Password: 
Last login: Wed. Jul 19 19:07:06: on console 


This computer system for authorized use only 


# ps -eaf 
UD PIDt PPID. -C SLIME EY TIME CMD 
root 0 Oe O22 49S 2 O15: sched 
root 1 O° *O;.2249'119.-2 O:00u/etc/init = 
FOOL 2 OED a OES? 0:00 pageout 
root 3 OY “09222-4950? 0:00) Fstlush 
root 22 is “02234945 ~--comso be” “OF 00m—sn 
root 173 Les Os 2249343702 0:00 /usr/sbin/syslogd 
root. 185 1 0° 22249344. 7 0:04 /opt/local/sbin/sshd 
LOOT L2h 212 "O..2253>42..console: 90.00 ps seat 
root 168 Ie 40°2203:4949.-2 0:00 /usr/sbin/cron 
df netstat -an 
UDP SEPv4 
Local Address Remote Address State 
erp Lae Idle 
eo Unbound 
TCP Leva 
Local Address Remote Address Swind Send-Q Rwind Recv-Q State 
+ % +e 0 O 24576 0 IDLE 
eee — 0 0 32768 0 LISTEN 
ie) Rae 0 O 32768 O: IDLE 
PCE IC PVo 
Local Address Remote Address Swind Send-Q Rwind Recv-Q State Jee 
e * Bae? 0 O 24576 0 TDLE 
# pkgchk -n 
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Appendix 2: Installation example 


Commands typed by the systems administrators are in fixed-width bold; comments are in italic roman 
typeface. 


zeta console login: root 
# ./install.sh 
Running YASSP installation script. 


nay 


Type the package list you want to install or hit return to accept the 
default: 


SECclean GNUrcs GNUgzip PARCdaily WVtcpd PRFtripw 
<return> 


We chose to install all packages proposed. 


Do you want to install SSH (See the SSH-COPYING file for the SSH 
lcense)?: [y|n] tn). 


We want SSH. 


YASSP will install: SECclean GNUrcs GNUgzip PARCdaily WVtcpd PRFtripw SSHsdi 
Installing the various package: 


aS SS SS SS SL SS 


SECclean 
SECclean installation start. 
The pre-install runs, initialize some variable and back-up files 
it will modify. 
Using /opty Vocal. as ‘the root-dir. 
Backing up all tiles under ./Vassp. bk/ Before 2000.,.07419-22 35 538 
Kiel. Ong. bis tO ta hes inc.) 
Pre-install is done. 
The install runs. files declared in the prototype are installed 
silently. Files, part of the sed class in the prototype, are 
modify by the associated sed script. 
Mod ity me Cc? 
The postinstall start. It first reads the variables stored by 
the pre-install. 
The postinstall script is silently running. It may take a while 
on slow machine. Just be patient 
Disabling init files we will replace later. 
Disabling startup files: inetsvc inetinit network 
Modifying startup files to be controlled by yassp.cony. 
Modifying. Startupersies to lise: etc, yassp cont =. (occ, busi of sal Startup 
fi hess moditded:o...7 
Creating /etc/yassp.conj, as we know now which startup file were modified. 
Creating your default /etc/yassp.conf 
Saving (in the package 5 save directory) and deleting files. 
Some of them will be replaced by SECclean s own version later. 
Savine  finlegs <5 dost of ti bes: backed ip 2247 
We have unregistered (removef) all the files we deleted from 
the package database. 
We must now close (removef -)) these open packages. 
Choosing the package we touched<; ..- package name l2st 4.7 
These are the files that will be replaced by SECclean 
version. They have been installed (as part of SECcleans 
prototype file) as /path/name/SECclean_{name_of_the_file}, and 
are registered under this name as part of SECclean package. We 
must first unregistered (removef on SECclean) them. 
Updating. (SbCc leancpackage DBs Ce dase of tile i? 
and close SECclean (removef -f SECclean) 
Closing SECclean DB 
Move the files from their SECclean_{name} to {name} and 
register them as part of SECclean (install) 
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Replacing (Cs. Lice Of tikes: a> 
OS specific startup files: chose the right version. 
Choesiiip thetright Startup tiless: fete/ingt.d/inetsve 
fete/inited/inetinit /etc/init.d/network-for your OS. Solaris 5.8 
Replacing them, registering (installf) as part of SECclean 
package, and creating the sym-link. 
Replacing Special ‘startup tiless/etc/inat .d/ inetsvc 
fetc/init.d/inetinit /etc/init.d/network and creating the symlink 
Closing (installf-f SECclean) SECclean. 
Closing again SECclean DB 
Specific OS tuning: for Solaris 8, no priority_paging 
Tuning /etc/system to comment out priority_paging 
Running clean_passwd 
Cleaning the passwd file... 
Dasabline. ULD. 0 account(s):: 
Disabling system account(s): 
daemon bin sys adm 1p uucp nuucp listen nobody noaccess nobody4 
Deleting account(s): 
root identity will be changed to "Root at zeta" 
password and shadow files saved under /etc/passwd.Old and /etc/shadow.0Old 
Doing the OS cleanup: fix-modes is run first... 
Doing the OS Clean-up 
Running tix-modes: “2:56:-2000/701/13 14513235. easper 
fix-modes done, log file under: /var/sadm/clean-up/clean_up.log 
Then we correct well-known incoherence in the Sun installation 
clean-up the contents database 
cleanup done, log file under: /var/sadm/clean-up/clean_up.log 
Running /usr/lib/makewhatis /opt/local/man 
We are done with SECclean, echo the summary. 








SECclean installation has finished. 
Changes to the file-system and package database are documented in: 
/var/sadm/clean-up/clean_up.log 
All changed or replaced files are archived in 
/yassp.bk 
If existing, the crontab for the users: 
lp adm uucp root 
has been deleted. Please, re-enable manually the entries needed 
Backup for the crontab files are under: 
/yassp.bk/var/spool/cron/crontabs/ 
To finish hardening, this host must be rebooted. 
However, you should first check that /etc/yassp.conf is configured 
to your requirements. See also yassp(1) and yassp.conf(4). 


—— SSS SS SS SS SS SSS SS SSS SSS SS SS SS SS SS SS SS se SS 











Installation of <“SECclean> was succeseful:. 


SSS SSS SS 


GNUrcs 


DeSean of <GNUrcs> was successful. 
cNUgzip 

eee of <GNUgzip> was successful. 
Sinead sie 

oe Geen eevee 

Installation of <PARCdaily> was successful. 


Eas 
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WVtcpd 
tcp wrappers add some example of how to use it in comment in /etc/inetd.conf 
Modifying ~ete/inet/inetd<cont 
and install some default configuration file ifthey were not present. 
Creating. /@te/ hosts deny fromthe distributron. tile 
*** Please configure it! 
Creating /etc/hosts.allow from the distribution. file 
*** Please configure it! 
iInstaliation.-or S<WViepd> was ‘successtul. 


Em et 


PRFtripw 


== SS ae Se 








Default configuration file is created if it was not present. 
Creating /secure/tripwire/tw.config trom the distribution: one; 
/secure/ tripwite/ tw.contip Dist 
*** Please configure it! 
you may use tripwire now. 


Types “edi fsecurey tripwire/: i/tripwire 1.2 -initialice =<c tweconfrie" 
to create a new database, 

Use “ed /secure/tripwire/; ../tripwire -q’-i 2 -e tw.config" 
to check, 


Pees OAVE. LOURY DATABASE, IN: A“SECURE: PLACE. ***"* 
installation of <PREttipw? was successtuls 


SSHsdi. 











Modifying /etc/syslop.cont 
default configurations files are created if they were not present. 
Creating fetc/ssh-contie from the distribution file 
*** Please configure it! 
Creating, /ere/sshd contig trom the distribution file 
*** Please configure it! 
We generate the keys iftheywere not present. 
Initializing random number generator... 
Cece 
Your public key has been saved in /etc/ssh_host_key.pub 
ssh has been installed. 
run” /ece/init.d/sshd -stop:/ete/init.d/sshd start and restart syslood 
to use the new binaries/configuration 
Installation of *CGSHsSdiv was successful 
YASSP installis done, now recreate the whatis database if it was present. 
Rebuilding the whatis database 
YASSP is installed. 
Most of these changes will take action at the next reboot. 
Pet YOUR: WORK 1 S.-NOE DONE, Yt 7-7" * 
*) Edit and scontipure /e6te/yasep cone 
*) Edie and configure “/ete/hostsi deny /ete/hosts..a low 
“) Edit. and -COnti pure v7ete/ sshd <contip sete/ssh-contip 
“) Read http: //yassp parc, xerox .com/atter.html 
and the: papers linked under http:/7/ Vassp. parc .xerox .com/ret html 
*) make any additional chanpes/scftware: installation 
*) CREATE YOUR tripwire DATABASE AND SAVE IT!!! 


Type: 
vie /6tC/yassp:.cont /ete/hosts-deny /ete/ hosts. allow./etc/sshd contig 
ferce/ssh.config +; cd /secure/tripwire; ./tripwire -i 2 -initialise -c 


tw. confieg) cp. /sdcure/tripwire/databases/tw.dbzeta TOA. SECURE PLACE 


***YOUR feedback*** is important: please send comments or flame to: 
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Ssansro@sans.org, chouanard@parc.xerox.com 
with. “YASSP™ ain “the subject 


#F reboot 
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ABSTRACT 


Internet security incidents have shown that while network cryptography tools like SSL are 
valuable to Internet service, the hard problem is to protect the server itself from attack. The host 
security problem is important because attackers know to attack the weakest link, which is 
vulnerable servers. The problem is hard because securing a server requires securing every piece of 
software on the server that the attacker can access, which can be a very large set of software for a 
sophisticated server. Sophisticated security architectures that protect against this class of problem 
exist, but because they are either complex, expensive, or incompatible with existing application 
software, most Internet server operators have not chosen to use them. 


This paper presents SubDomain: an OS extension designed to provide sufficient security to 
prevent vulnerability rot in Internet server platforms, and yet simple enough to minimize the 
performance, administrative, and implementation costs. SubDomain does this by providing a least 
privilege mechanism for programs rather than for users. By orienting itself to programs rather than 
users, SubDomain simplifies the security administrator’s task of securing the server. 


This paper describes the problem space of securing Internet servers, and presents the 
SubDomain solution to this problem. We describe the design, implementation, and operation of 
SubDomain, and provide working examples and performance metrics for services such as HTTP, 
SMTP, POP, and DNS protected with SubDomain. 


Introduction 


Common server operating systems such as 
Linux, Windows, Solaris, etc. are subject to vulnera- 
bility rot as security vulnerabilities (i.e., implementa- 
tion bugs) are discovered in the component software 
of these operating systems. For instance, a buffer over- 
flow discovered in the BIND domain name server [15] 
allowed remote attackers to gain root privileges on a 
variety of system platforms, and a similar vulnerabil- 
ity in Microsoft’s IIS (web server) [21] allows remote 
attackers to gain administrative control of Windows 
servers. The recommended defense for general pur- 
pose servers is to keep the host system up to date with 
vendor patches to close these vulnerabilities. 


However, many of these systems are being 
pressed into use as the basis for server appliances: 
servers intended for largely unattended operation by 
unskilled users. But because these operating systems 
are subject to vulnerability rot, they need to be fre- 
quently upgraded with vendor patches. While this is 
an acceptable approach for general purpose servers 
(where a skilled system administrator is expected to 
maintain the system) it is not acceptable to appliance 
users, who expect a device with the maintenance fac- 
tor of a toaster. 


‘Cowan, Beattie, and Pu are formerly of the Oregon Grad- 
uate Institute, where much of this work was done. Cowan 
and Beattie are now with WireX Communications, and Pu is 
now with the Georgia Institute of Technology. Wagle is still 
with the Oregon Graduate Institute, and Gligor is with the 
University of Maryland. Kroah-Hartman is with WireX. 
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The classical security solution to vulnerability 
rot is the notion of /east privilege: the technique of 
granting subjects in a system precisely the capabilities 
they need to perform their function, and no more [33]. 
Effective use of least privilege minimizes the potential 
damage that results when a trusted program is pene- 
trated by minimizing the degree to which the program 
is trusted. 


Security architectures that provide least privilege 
mechanisms exist, but because they are either com- 
plex, expensive, or incompatible with existing applica- 
tion software, appliance vendors have not chosen to 
use them. Existing defenses entail these complexities 
precisely because they were designed to handle the 
generality of a general purpose server, and thus must 
deal with user least privilege. This generality compli- 
cates the least privilege abstraction, making the 
enforcement mechanism more complex to implement 
and use. 


This paper present SubDomain: an OS extension 
designed to provide sufficient security to prevent vul- 
nerability rot in server appliances, and yet simplify as 
much as possible to minimize the performance, 
administrative, and implementation costs. SubDomain 
does this by providing a least privilege mechanism for 
programs rather than for users. The security restric- 
tions complement the system’s existing permissions, 
allowing a program to be secured independent of who 
may be using the program. This notion is especially 
effective on server appliances, and enables program- 
specific confinement information to be distributed 
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with the program (see the section on SubDomain com- 
patibility). 

By specifically addressing least privilege for pro- 
grams, we can provide a mechanism that has a rela- 
tively small implementation and simple operation. 
Small implementations are important for security sys- 
tems to avoid vulnerabilities due to bugs in the 
enforcement mechanism itself. Simple operation 1s 
important for security systems to avoid misconfigura- 
tion. Even more so than in most OS design issues, par- 
simony is critical to security [33] making SubDo- 
main’s relative simplicity of design and implementa- 
tion an important feature. 


We present the SubDomain notation for recur- 
sively specifying the sub-domain of resources avail- 
able to a software component, our implementation of 
SubDomain as an enhancement to the Linux kemel, 
our application of SubDomain confinement to several 
example applications, performance metrics on the cost 
of SubDomain confinement, and our analysis of the 
security of a system protected by SubDomain. 


The challenge of supporting least privilege is to 
provide a specification system that is expressive 
enough to specify privileges that are actually minimal, 
is convenient enough that administrators can reason- 
ably specify least privileges, and yet preserves com- 
patibility and performance. While SubDomain strives 
for simplicity relative to other least privilege mecha- 
nisms, it provides for finer granularity least privilege 
in one important regard: SubDomain can confine arbi- 
trary software components, at a finer granularity than 
the native OS process, 1.e., procedures and modules. 
This 1s especially important for component-based ser- 
vices such as Apache [9] and its loadable modules (see 
the section on confinement). 


The rest of this paper is organized as follows. 
The next elaborates on the problem of vulnera- 
ble/buggy software, and describes the abstract solution 
of least privilege to minimize the potential damage 
due to attacks against vulnerable software. Readers 
familiar with least privilege can skip ahead to the third 
section, which describes the SubDomain security 
enhancement, and how it advances over previous least 
privilege mechanisms by providing finer granularity, 
and simplifying the problem of confining suspect pro- 
grams. The fourth demonstrates SubDomain’s com- 
patibility by confining assorted software components, 
including sub-process modules. The fifth section pre- 
sents the performance costs of SubDomain confine- 
ment. The subsequent section 6 describes related work 
specifically addressing the problem of confining sus- 
pect programs. The final section presents our conclu- 
sions. 


The Problem: Vulnerable Programs and Least 
Privilege 


Many security vulnerabilities result from bugs in 
“trusted” programs. A “trusted program” is a 
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program that runs with privilege that some attacker 

would like to have, and the program fails to keep that 

trust if there 1s a bug in the program that allows the 
attacker to acquire that privilege. Some examples 
include: 

Buffer Overflows: Many privileged programs con- 
tain “‘buffer overflow” vulnerabilities, a prob- 
lem endemic to C programs that provide poor 
bounds checking on user-supplied input. Buffer 
overflows are very common [18, 19] and very 
dangerous [32, 29], allowing attackers to take 
control of programs from an anonymous node 
on the internet. 

Race Conditions: Many privileged programs also 
contain “‘race condition” vulnerabilities. Here, 
the problem is that careless root privileged pro- 
cesses create files without adequate checking 
for the prior existence of the file. The problem 
is that the attacker can create a symbolic or 
hard link in the file system between the time the 
privileged program checks for existence and the 
time it creates the file, with the result that the 
root program unwittingly uses its authority to 
corrupt some other critical file [12]. 

Special Character Processing: While few root 
privileged programs are written in shell script- 
ing languages, many other programs with 
“interesting” privileges are written as shell 
scripts, especially CGI/ PERL [41] programs 
for processing web forms. CGI programs run 
with the authority of the web server, and must 
process arbitrary input from arbitrary users. If 
the attacker can provide input (using creative 
URLs) to a CGI program that yields control to 
the attacker, then the attacker can gain control 
of the web server, e.g., the PHF program 
(included in early NCSA and Apache web 
servers) allowed the attacker to present a URL 
to the web server that would cause PHF to start 
an xterm on the attacker’s display [14]. 

Note that while “trusted’’ usually refers to highly 

privileged processes (e.g., root processes) they can 

actually be processes with any privileges that the 
attacker wants but does not have. The general case is 
that any program installed on a computer that pro- 

cesses input from potentially hostile users becomes a 

potential vulnerability. Eliminating these vulnerabili- 

ties requires some form of assurance that the program 
in question does not contain exploitable bugs, but this 
kind of assurance is problematic. Some classes of 
bugs, e.g., buffer overflow vulnerabilities, can be 

eliminated through various compiler techniques [39, 

17, 26, 37]. Other forms of vulnerabilities are unde- 

tectable at compile time, e.g., race conditions [12] and 

general logic errors. 


The only way to assure the complete absence of 
a security vulnerability in a program is through expen- 
sive manual verification. In the absence of such verifi- 
cation, one must either suffier the risk of potential vul- 
nerabilities, or contain the potential damage. Note that 
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the activities we seek to constrain are “those that 
cause damage to the system,” 1.e., safety properties 
[1] with respect to integrity. We are not addressing 
other security issues, such as information flows [27] 
that might disclose secrets. Readers already familiar 
with least privilege mechanisms can skip to to the next 
section for a description of SubDomain, our contribu- 
tion to the field. 


The Solution: Least Privilege 


The classic solution to the problem of unknown 
security vulnerabilities is to perform each activity with 
the least privilege required to complete that task [33]. 
While this does not stop exploitation of these vulnera- 
bilities, it does contain the damage as much as possi- 
ble. An attacker who gets control of a least privilege 
process can, at most, read secrets and corrupt data that 
the exploited process has access to, and no more. 


The challenge of supporting least privilege is to 
provide a sufficiently fine-grained mechanism to spec- 
ify privileges that are actually minimal, while also pre- 
serving compatibility and performance. It is conceptu- 
ally simple to divide system privileges into fine- 
grained units and then attribute the exact required 
privileges to a given activity, but the result of such an 
approach is specification notation that is tedious to 
maintain (breaking compatibility) and an enforcement 
mechanism that is slow (breaking performance). 


Practical least privilege therefore involves 
abstracting the system resources to expedite least priv- 
ilege specifications. Matching least privilege abstrac- 
tions to native OS resources in turn enables efficient 
least privilege enforcement. Least privilege is also a 
useful notion in managing user privileges, leading 
many systems to combine least privilege for users and 
programs into a single mechanism, as described in the 
next subsection. 


However, if the problem is bugs in programs that 
can be accessed by completely untrusted users, then 
user-oriented least privilege mechanisms may become 
awkward or inadequately expressive. The section on 
users and rolls describes some more elegant 
approaches to using user privilege mechanisms to con- 
fine suspect programs. The third section discusses 
SubDomain, our OS security enhancement that partic- 
ularly address the problem of least privilege for pro- 
grams, and a penultimate section discusses related 
work specifically aimed at program confinement. 


Using User Privileges to Confine Programs 


Least privilege for users is a classic way of struc- 
turing a system, and many operating systems provide 
facilities for constraining the privileges of a given 
user. User-oriented least privilege facilities can be 
adapted to confining collection of programs by creat- 
ing a synthetic user, and then running the program as 
that user. 


The classic example is the UNIX setuid facility: 
the setuid bit for an executable file indicates that the 
program runs with the privilege of the owner of the 
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file instead of the privilege of the invoking user. Often 
this is used to create setuid root programs that provide 
controlled access to protected resources by expanding 
the privileges the program runs with to be all of root’s 
privileges. To use setuid to confine a program to a 
smaller set of resources, a new synthetic user can be 
created that has those privileges, e.g., nobody. Pro- 
grams can then be made setuid nobody to confine their 
actions to a small set of privileges. 


One limitation to this approach is that all user- 
IDs, even synthetic user-IDs, can access all files on 
the system that permit “‘other’’ accesses. Another lim- 
itation to this approach is that only root can create new 
user-IDs. The result is that normal users cannot con- 
struct ad hoc “‘sandboxes”’ for programs that they may 
choose to install and run. Users are then left with their 
choice of: 
e beg the system administrator to create a new 
user-ID for them, 
e do not install software that is not trusted, 
e run untrusted software without protection, none 
of which 1s very appealing. 


So in principle, synthetic user-IDs and the setuid 
mechanism can support least privilege for programs, 
but in practice it forces root to do all the work. There- 
fore this technique is rarely deployed, people run un- 
trustworthy software with much more privilege than 1s 
necessary, and suffer the consequent security risks. 


Users and Roles 


Because synthesizing user-IDs is awkward, the 
notion of a role emerged. A role is a collection of 
related privileges [2]. In 1986, Bobert and Kain intro- 
duced the notion of type enforcement: objects (files) 
are assigned to types, subjects (processes) are assigned 
to domains, and tables determine which domains have 
access to which types. Badger et al expanded on this 
notion [7, 8]. In a similar vein, role-based access con- 
trol (RBAC) [22, 34] assigns users to roles, and then 
grants privileges to the roles. 


Similar to the setuid approach described previ- 
ously, roles can be pressed into service confining pro- 
grams to a least privilege set of resources by assuming 
a specific role just prior to executing the program. 
While using roles to confine programs is more elegant 
than synthesizing user-IDs, it is still fundamentally 
overloading a user-oriented access contro] mechanism 
to manage software defects. In the next section, we 
describe our mechanism to specifically address the 
problem of vulnerable software. 


SubDomain Security: Recursive Component Con- 
finement 


SubDomain is a kernel extension designed 
specifically to provide least privilege confinement to 
suspect programs. SubDomain allows the admuinistra- 
tor to specify the domain of activities the program can 
perform by listing the files the program may access, 
and the operations the program may perform. 
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SubDomain restrictions complement the native access 
controls, in that SubDomain never expands the set of 
files a program may access, 1.e., any file access must 
pass the native access controls and the SubDomain 
restrictions before access 1s granted. Thus SubDomain 
confinement makes a program monotonically safer to 
run. 


The next subsection describes the SubDomain 
notation and semantics. Then, the subsequent subsec- 
tion explains how SubDomain leverages work in safe 
programming models like proof-carrying code [30] to 
achieve component confinement below the granularity 
of a native process. The final subsection describes the 
SubDomain implementation. 


SubDomain Notation & Semantics 


Figure 1 shows a trivial SubDomain specifica- 
tion, in which the foo program is given read access to 
the /etc/readme file, write access to the /etc/writeme 
file, and execute access to the /usr/bin/bar file. When 
ever the program foo 1s run, by any user, it 1s restricted 
to access these specified files with these modes. Sub- 
Domain profiles can also grant access to directories 
through simple globbing, 1.e., the profile in Figure 1 
grants the foo program to all files in /mydir. 


foo, { 
/etc/readme 
/etc/writeme 
fusr/ban/ bar 
Jnydars* 

} 


RM ER 


Figure 1: Trivial subDomain. 


£60. 
/etc/readme rR 
/etc/writeme w=, 
jusr/bin/bar “«-(/ ete/otherwrite wl = 
/usr/bin/baz x -{/etc/writeme wh 4 


} 
Figure 2: Relative SubDomain 


The X (execute) capability is of particular impor- 
tance: what restrictions should apply to the child pro- 
cess? By default, the child process inherits the parent’s 
SubDomain, preventing the confined program from 
“escaping” its confinement by executing an unre- 
stricted child process. However, sub-components of 
an application may require different capability sets 
than the application as a whole. For instance, games 
only need strong privileges to initialize video con- 
trollers, and mail delivery agents only need strong 
privileges to actually write to a user’s mail box. Thus 
child programs can be given different constraints by 
specifying a relative subdomain, denoted by a X fol- 
lowed by a + or - followed by a SubDomain specifica- 
tion. For example, Figure 2 shows a SubDomain for 
foo that says that when the sub-component bar is run, 
it can also have write permission to the /etc/otherwrite 
file. Conversely, it says that when foo runs the sub- 
component baz, it may not write to the /etc/writeme 
file. 
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Sub-components may also want a SubDomain 
that is completely unrelated to the parent domain. For 
example, a web server application might need to send 
some e-mail while processing a web form, and thus 
invokes a mail delivery agent whose SubDomain is 
completely different. We support this need with abso- 
lute subdomains, denoted by a subdomain specifica- 
tion following an X without a + or a-. Figure 3 shows 
an example absolute subdomain in which the bar pro- 
gram run from the foo program has access to a com- 
pletely different subdomain than the foo program. 


£oo- =| 
/etc/readme r, 
/etc/writeme We oe 
/usr/bin/bar x { 


fisr/ltb/otherread. “1-4, 
[var/opt/otherwrite: w 5 


Figure 3: Absolute SubDomain. 


When a confined process tries to perform a file opera- 
tion that 1s not permitted, two things happen: 

1. The syscall returns with the error EPERM, just 
as if the attempt had failed due to a standard 
UNIX file system permission error. 

2. The kernel generates a syslog entry describing 
the attempted violation. Intrusion detection 
systems can thus collect what ever information 
they want, and act accordingly, e.g., kill the 
offending process if such drastic steps are 
desired. 


Sub-process Confinement 


The section on related work describes several 
other systems that provide program-confinement 
mechanisms. However, with the exception of Java [4] 
the smallest component that they can confine is a 
native OS process. In contrast, SubDomain provides 
the unique feature of being able to confine compo- 
nents that are only a portion of an OS process. Histori- 
cally of little practical interest, the need for sub-pro- 
cess confinement comes from the rise in popularity of 
scriptable servers and loadable modules. Let us 
expand upon these concepts. 


A “scriptable server” 1s a server program that, 
from time to time, interprets a script or a program 
within itself, i.e., server-side includes [5], PHP web 
pages [6], Java servlets [3], etc. Such scripts are legiti- 
mately sub-component programs requiring separate 
confinement. Scriptable servers often have their own 
security mechanisms, but in depending on _ such 
restrictions, we are depending on application correct- 
ness, which 1s the dependence we seek to avoid in the 
first place. We would rather have a confinement mech- 
anism that can be enforced by the operating system so 
that we do not depend on the correctness of the server 
application. 


‘““Loadable modules” or ‘‘plug-ins”’ is the notion 
of providing a (fairly) fixed API in an application so 
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that extensions to can be loaded into the application, 
either at start-time or run-time. “Plug-in” is the com- 
mon term for desktop applications (i.e., Netscape Nav- 
igator & Shockwave, Microsoft Word and EndNote) 
while “module” is the common term for servers (e.g., 
Apache and mod_perl). 


T he mod_perl module for Apache provides a per- 
fect example of the sub-process problem. PERL 
scripts run at the behest of the Apache web server are 
normally interpreted by starting a separate process to 
run the PERL interpreter, and then interpreting the 
PERL script in that separate process. mod_perl loads a 
PERL interpreter directly into the Apache process to 
avoid the cost of starting the PERL interpreter process. 
While this is good for server throughput, it 1s bad for 
security: 

e Bugs in mod_perl can crash the Apache web 
server process. 

e Program-confinement mechanisms that only 
operate on OS processes cannot confine scripts 
interpreted by mod_pen separate from the 
Apache web server process. 


The SubDomain solution to the “scripting & 
module” problem is to provide for subdomains for 
sub-process components, in cooperation with the 
enclosing application. The notation for a sub-process 
subdomain is unchanged from that of separate-process 
subdomains shown in Figure | through Figure 3. The 
effect is to create a variety of “‘hats” that process can 
wear, one for each sub-process component that it calls. 
The “‘cooperation” required from the enclosing pro- 
gram is that it should call the new change_hat() system 
call before calling the sub-process component. 


The requirement to call change_hat() implies that 
we are once again trusting the application, which Sub- 
Domain 1s supposed to avoid. However, we are trust- 
ing the application code a great deal Jess, in that the 
application only has to make appropriate calls to 
change_hat(), which is much simpler than constructing 
and enforcing an effective “‘sandbox’’ environment 
[20]. Successfully calling change_hat() with the name 
of a sub-component before calling the sub-component 
seems easy enough to do correctly. 


In addition to the requirement that enclosing 
application correctly calls change_hat(), we also 
require that the sub-component does not call 
change_hat() to escape to a more liberal subdomain. 
Here, we employ a cookie argument to change_hat() to 
prevent the confined module from escaping. The con- 
taining process initially calls change_hat() with a par- 
ticular cookie value, and further change_hat() calls that 
do not provide a matching cookie argument are treated 
as security violations. 


Thus for the containing process to prevent sub- 
component from escaping from the change_hat() Sub- 
Domain, it need only provide a cookie value that the 
contained sub-component cannot easily guess. We rec- 
ommend fetching a word from /dev/random, but any 
reasonable source of entropy can be used. 
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The security of this method depends on the sub- 
component not being able to read the parent process’s 
cookie value. Here, SubDomain can leverage the 
power of language-based security protection systems 
such as proof-carrying code [30], strong type checking 
[39, 26, 37], and other language-based protection 
schemes [24, 40]. Such methods can, in principle, 
prove that the sub-component will not invoke the 
change_hat() system call. 


Programming language techniques provide pow- 
erful protection, but also impose significant practical 
constraints, not the least of which 1s that the sub-com- 
ponent needs to be written in a particular language. In 
practice, we can still get reasonable assurance that the 
sub-component cannot read the containing process’s 
cookie value if it 1s written in a scripting language, 
1.e., a language that 1s interpreted rather than one com- 
piling to native CPU instructions. In the practical set- 
ting of scripts for web servers, most such programs 
that are executed by loadable modules are scripting 
languages, e.g., PERL [41], PHP [6], and Java [3]. 
While no formal assurances are available, in practice it 
is easy to trust, say, mod_perl to not address random 
memory. 


To see the power of this approach, consider the 
chronic problem of securely supporting Microsoft’s 
‘Front Page Extensions,” a collection of non-standard 
HTML tags that the server interprets to provide more 
dynamic HTML content. Microsoft provides a mod_fp 
Apache module and collection of helper programs, but 
they have a poor security history [36]. There is no cur- 
rent practical method to securely support mod_fp. 


SubDomain can solve the mod_fp problem by 
treating web pages containing “Front Page Exten- 
sions”’ tags as sub-components, and assigning each 
such page to a subdomain. So long as the mod_fp mod- 
ule can be trusted not to call the change_hat() system 
call, then no errant action of mod_fp can violate the 
security policy of the subdomain for the page it 1s 
interpreting. 

SubDomain Implementation 


The basic architecture of SubDomain is shown in 
Figure 4. The SubDomain policy engine is imple- 
mented as a Linux [38] loadable kernel module. Fol- 
lowing the usual UNIX permissions checking, the rel- 
evant system calls (open(), exec(), read(), etc.) are 
modified to check if the calling process is a confined 
process. If so, the request is referred to the SubDo- 
main module for further inspection. The SubDomain 
module then either returns normally (if the request 1s 
permitted) or retums an EPERM error (if the request is 
denied). 


Once loaded, the SubDomain module disables 
module unloading to prevent tampering with the Sub- 
Domain policy engine. A user-level parser reads sub- 
domain profiles from /etc/subdomain.d/* to convert 
the textual representation of profiles into kernel data 
structures, and inserts the updated profiles into the 
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kernel via a sysctl) interface. By convention, the 
/etc/subdomain.d/foo file would confine the foo pro- 
gram, but as shown previously, the actual name of the 
confined program 1s in the file, so confining multiple 
components with a single file 1s possible. Only root 
processes can access this kernel interface, and SubDo- 
main-confined programs may not access the profile 
interface. In future work we will add further authenti- 
cation requirements to the kernel’s profile interface. 


SubDomain Parsimony 


SubDomain 1s simpler than competing least priv- 
ilege mechanisms described in the section on related 
work in both implementation and usage. With regard 
to implementation, the SubDomain module and kernel 
patches amount to 4500 lines of C code, and the non- 
kernel parser is 825 lines. In contrast, the DTE kernel 
enhancement [7, 8] 1s over 40,000 lines of code. The 
relatively simple semantics of SubDomain enable a 
smaller implementation. ‘Smaller’? 1s important for 
security systems, where correctness 1s critical, because 
bugs are approximately liner in code size. 


SubDomain’s usage is simpler than its competi- 
tors in that it is easier to devise and inspect SubDo- 
main confinement profile than in other systems, which 
we elaborate on in the next section. 


SubDomain Compatibility 


We test the compatibility of SubDomain by 
putting it to work confining a variety of software com- 
ponents common to Internet servers, both large and 
small. SubDomain can confine binary-only programs, 
so long as there is no need for sub-process confine- 
ment. If sub-process confinement is required, then the 
program source needs to be edited and re-compiled to 
insert appropriate calls to change_hat(). 

Like the “synthetic user”? approach mentioned 
earlier, SubDomain confinement requires admuinistra- 
tor intervention. However, SubDomain confinement is 
easier for the administrator in the following ways: 
Ease of Application: A SubDomain profile does 

not interfere with any other aspects of the sys- 

tem except the SubDomain mechanism. Thus it 


/etc/subdomain.d/* 


File acces¢ 
request 
e.gopen ( ) 
read () 


Laccess contr®is SubDomain | Accept 


Cowan, et al. 


is easy to install a SubDomain profile along 
with the confined program. In_ particular, 
because the SubDomain profile is independent 
of the system the program 1s being installed on, 
the profile can be included with the program 
being distributed. In contrast, it is difficult to 
include a synthetic user in conventional pro- 
gram packages (e.g., tar balls or RPM 
packages). 

It is easy for the administrator to inspect a Sub- 
Domain specification to determine the precise 
aspects of the system that are exposed to that 
program. In contrast, the exposure entailed by a 
synthetic user is non-obvious: the administrator 
must consider all files that are accessible to 
“anybody,” which is a non-trivial exercise on 
non-trivial file systems. 


The Kernel Wrapper approach [23] provides for 
confinement scripts that are full Turing-equiva- 
lent programs. While this provides extensive 
flexibility, it also means that the completeness 
and safety of an inserted kernel wrapper is not 
amenable to automatic analysis. In contrast, 
SubDomain profiles are easy to inspect to 
determine the security implications of updating 
a SubDomain profile. Furthermore, it 1s strictly 
safe to install a SubDomain profile where none 
existed before, because SubDomain strictly 
limits program privileges. 


These factors have important implications for 
software distribution. Because SubDomain confine- 
ment profiles are system independent and guaranteed 
to be safe to install, it becomes feasible to package 
SubDomain confinement with the program itself. Thus 
an end user can consider installing a new program on 
a server appliance, and because of the SubDomain 
confinement information packaged with the program, 
the user can understand the security implications of 
installing that program. In future work, we plan to 


2Note that bundling synthetic user IDs is exactly the ap- 


proach taken by qmail [11], which results in excellent se- 
curity for qmail, but also imposes substantial packaging 
difficulties that have hampered qmail’s spread. 







Parser 








Module 


Figure 4: SubDomain implementation 
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develop future tools that will assist the administrator 
in determining the security implications of a set of 
SubDomain confinements 


Which programs need to be confined with Sub- 
Domain depends on the convenience and security 
needs of the host system, and thus 1s an adjustable pol- 
icy. The administrator can specify which of the fol- 
lowing classes of programs must be confined with 
SubDomain before they are allowed to execute at all: 
All Programs: All programs that execute on the 

host must be associated with a SubDomain, 

either explicitly, or inherited from a SubDo- 
mained parent program. This mode is suitable 
for bastion hosts. 

All Listed User-IDs: All programs running under 
one of the user-IDs specified by the administra- 
tor must be associated with a SubDomain. For 
instance, the httpd user-ID runs many programs 
on behalf of the web server, and SubDomain 
confinement ensures that these programs will 
not affect other parts of the system. This mode 
is suitable for confining a potentially vulnerable 
collection of services on a system that also 
hosts critical data. 

All root Programs: All programs running with a 
real or effective user-ID of “‘root.”’ This mode 
allows a SubDomain profile to be used to 
achieve the classic goal of breaking up root’s 
all-too-powerful privileges. The (defunct) 
POSIX 1.e “capabilities” model subdivided 
root’s powers into a static set of 32 separate 
groups of “capabilities”, and individual pro- 
grams could assume part of root’s powers by 
flipping on one or more of these sets of capabil- 
ities. SubDomain allows arbitrary sets of privi- 
leges to be grouped together, rather than accept- 
ing the groupings specified by POSIX 1.e. 

Only Specified Programs: Only the programs that 
have a SubDomain specified are thus confined, 
1.e., “default allow.’’ This mode assumes that 
all programs on the host are adequately secured 
except for the programs being SubDomained. 
While not especially secure, this mode is con- 
venient, e.g., for use on a client workstation to 
run a suspect program recently downloaded 
from the Internet. 


The procedure for confining a program 1s to start 
with a null subdomain specification, run the applica- 
tion, observe the system log for complaints about 
attempts to access files outside the subdomain, and 
then add those files to the subdomain specification. 
This procedure is presently manual, because due con- 
sideration is required for two stages in this procedure: 
Running the application: The application needs to 

be run under all of the “kinds”’ of input that it 

1s expected to experience in a production envi- 
ronment, i.e., a comprehensive test suite. Deter- 
mining these inputs requires some knowledge 
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of the application to ensure complete coverage. 
Failure to provide complete coverage results in 
a subdomain that is too “tight”, and the appli- 
cation will occasionally fail to access resources 
that it needs. 

Granting the privilege: We are confining the appli- 
cation precisely because we do not trust it, and 
therefore we cannot automatically assume that 
every file the application tries to access under 
test is a legitimate file for the program to 
access. The file should be included in the sub- 
domain only after due consideration of the 
security implications. 

For applications where source code is available, pre- 

dicting the set of required resources should be feasi- 

ble. If anticipating the set of files an application needs 
to access is truly difficult, then it is quite likely that 
the application represents a significant security threat, 
and should not be installed on hosts requiring security. 


For applications where source code is not avail- 
able, a run-time testing methodology must be used to 
experimentally identify all of the file resources that a 
program may try to access. To facilitate this, we use 
the dep program that we developed for the InDepen- 
dence project [16] (funded by a student grant from 
USENIX). This program uses strace() to monitor the 
execution of a subject program, and amasses a list of 
all the files accessed. dep’s use of strace() imposes 
heavier performance and compatibility overhead than 
SubDomain, but is none the less sufficient for explor- 
ing the file system domain of many programs. To fur- 
ther ease use, dep accumulates files accessed across 
multiple runs, so that a large test suite can be applied, 
and then the list of files accessed inspected once at the 
end of testing. 


/home/httpd/cgi-bin/Count.cgi 
/etc/1ld.so.cache 


r 
Libya bs r 
/lib/1d-linux.so.2 rc 
/etc/nsswitch. conf r 
/etc/wwweounter.conf r 
/etc/localtime rm 
/var/log/httpd/wwwcount/wwweount_log rw. 
/var/lib/wwweount/* Cc & 
/var/lib/wwwcount/data/* rw , 


) 
Figure 5: SubDomain for wwwcount CGI script 


An example subdomain profile is shown in Fig- 
ure 5, providing all of the resources needed to run the 
wwwcount CGI program (a popular web page hit 
counter program). Note the use of simple globbing to 
reduce the size of the subdomain specification when 
access to an entire directory is required. Figure 6 
shows a more elaborate profile for the Apache web 
server itself, under a particular configuration. A list of 
some of the programs that we have confined and 
tested, along with the size of their subdomains, are 
listed in Table 1. 
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/usr/local/apache/bin/httpd { 
/ 


Ft 


/dev/null 

/dev/urandom 

/etc/group 

/etc/hosts 

/etc/host.conf 
/etc/1d.so.cache 
/etc/localtime 
/etc/nsswitch.conf 
/etc/passwd 

/etc/resolv.conf 
/home/httpd/perl/* 

(Aa 

/usr 
/usr/1ib/gconv/1IS08859-1.so 
/usr/lib/gconv/gconv-modules 
/usr/1ib/per15/5 .00503/* 
/usr/1lib/per15/site_perl/5.005/1i386-1linux/* 
/usr/local 

/usr/local/apache 
/usr/local/apache/conf/* 
/usr/local/apache/htdocs/* 
/usr/local/apache/logs* 
/usr/share/locale/en_US/* 
/usr/share/locale/locale.alias 


4 
= 


HAR SAAN PARA TREN OR OO OO OA OA 
he 
- 


Figure 6: SubDomain for Apache Web Server. 


SubDomain Performance 


Here we present a variety of SubDomain perfor- 
mance measurements. The next section describes our 
microbenchmarks on mediated system calls, and the 
section after that describes our macrobenchmarks on a 
confined PERL script interpreted by the mod_perl 
Apache module. 


Microbenchmarks 


Here we use the usual benchmarking technique 
to measure affected system calls by crafting programs 
that issue each system call 10,000 times, run the pro- 
grams several times, discard the first run to avoid cold 
cache effects, and average the remainder. All tests 
were performed on a dual-processor Pentium III 700 
MHz, with 256 MB of RAM. Table 2 summarizes 
these results. We include measurement of the get_pid() 
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system call as a baseline for comparison against the 
change_hat() system call, as get_pid() is commonly 
regarded as the simplest system call. 


As expected, he major overhead appears in the 
open(), exec() and change_hat() system calls, where 
SubDomain is checking the action against the subdo- 
main specification for the confined process. 


Size of SubDomain 


eS 16 files 


10 files | 
loc 11 files | 


Postfix Mail Delivery Agent 15 files | 
Postfix-script helper program 65 files | 


Table 1: SubDomain-confined programs. 





Macrobenchmarks 


Our macrobenchmark is SubDomain confine- 
ment of a PERL script to be executed via the mod_perl 
Apache module, thus exercising SubDomain’s capa- 
bility to confine active content scripts. To exercise the 
web server’s cache, we replicated the PERL script 
1000 times, and used the Webstone performance 
benchmark to measure the overhead cost of PERL 
scripted web pages protected with SubDomain vs. 
without protection. The PERL script itself reads two 
files with some busy-work in between, simulating a 
script that fetches a “container” template from one 
file, HTML content from another file, and does some 
interim processing to merge the two, e.g., compute a 


| System Call Standard Cost SubDomain Cost | “% Overhead 


ok) 
(387 [487 


end rar 
get_pid() vs. change_hat() | 1.81 159% 


Table 2: SubDomain Microbenchmarks in microseconds. 





/perl1/0/cgitest-O00l.cgi { 


/usr/lib/per15/site_perl/5.005/i386-linux/Apache/Registry.pm 


fetc/localtime 
/usr/ li b/ perl5/500503/* 
/home/httpd/perl/0/cgitest-001.cgi 


/home/httpd/perl/0/cgitemplate-001.html 


/home/httpd/perl/0/cgidata-001 
/var/log/httpd/* 


Si eit ech, SES is: - SES 


Figure 7: Test PERL script’s SubDomain profile. 
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hit counter. The SubDomain profile for this script 1s 
shown in Figure 7. 


The test environment used the same dual-proces- 
sor Pentium III 700 MHz server with 256 MB of 
RAM, and a private network (crossover cable) via 100 
Mbit ethernet. 


The test results are shown in Table 3, measured 
for 5 to 10 concurrent client connections. Tests were 
run twice, and the results averaged. For all cases, the 
SubDomain overhead is between 1% and 2%, i.e., in 
the noise range. 


Related Work 


Here we describe work that, similar to SubDo- 
main, specifically attacks the problem of confining 
suspect programs. Despite the age of the notion of 
least privilege [33], much of this work has emerged 
relatively recently. It is our conjecture that this is a 
result of a shift in emphasis from defending secrecy 
(the dominant concern for military organizations) to 
defending integrity (the dominant concem for Inter- 
net-connected businesses) and the emergence of the 
notion of survivability [35]. This list of related work is 
necessarily partial, as the total body of related work is 
very large. 

TRON 


The TRON system [10] 1s a kernel enhancement 
for ULTRIX that can confine a program’s execution to 


Test # of Connection 
Clients Rate 

Standard ; 5 | 75.97 

‘SubDomain | _s | _ 75.19 
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a protection domain consisting of a finite set of capa- 
bilities in the form of file names. TRON adds the 
tron_fork() system call, which functions exactly like 
the classic fork() system call, except that it specifies 
the protection domain as an extra argument. TRON is 
semantically most similar to SubDomain: the protec- 
tion domains are the same (sets of files) and are simi- 
larly applied to host programs, orthogonal to user priv- 
ileges. The major differences are: 

e TRON is discretionary, while SubDomain is 
mandatory. TRON provides user commands to 
run programs in a confined domain, while Sub- 
Domain always runs a specified program in a 
confined domain. Thus in the usual DAC vs. 
MAC trade-off, TRON is more convenient for 
individual users, while SubDomain is more 
convenient for securing entire systems, e.g., 
server appliances. 

TRON’s finest granularity is the ULTRIX pro- 
cess; it cannot confine loadable modules. 


Janus 


Janus [25] is a user-level mechanism for confin- 
ing programs to a specific set of resources. Intended to 
confine “helper” applications run from within a Web 
browser, Janus uses the ptrace() system call and a 
monitoring process to mediate all system calls made 
by the helper application. If the action proposed by the 
helper application violates a policy set by the user, 
then the monitoring process rejects the request. This 


Avg. Response | Avg. Client 
Time (ms) Throughput 
66.5 26.29 


26.02 


% Overhead eh 


“Standard 


SubDomain 
-% Overhead | 


Standard 
SubDomain 





“ 


‘ Overhead | 1.45% 
;__ 8 | 78.26 102, 27.08 
8 76.71 104 | 26.54 


Standard 
SubDomain 


78 38 
77.24 


— 56 26.49 
2% —39 —ae 


emanate 
3 ee 0S I 26.73 | 
1.7% | 1.5% 


% Overhead 2% 2% 


‘Subbonais [S| 77 1165 | 26.6 


% Overhead 
10 


Standard 
SubDomain 


% Overhead 


Table 3: SubDomain macrobenchmarks with WebStone. 
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approach requires four system calls to be executed to 
effect one confined system call. 


Java 2 Security 


The Java 2 security model [4] allows the JVM to 
be configured to assign particular capabilities to desig- 
nated Java classes, similar to the SubDomain notion of 
assigning file system capabilities to programs. This is 
an enrichment over the original Java security model 
[26] which assigned one fixed set of capabilities to 
remotely-loaded applets (almost nothing), and another 
fixed set of capabilities to locally-loaded applets 
(almost everything). 


The Java 2 security mechanism is notable as the 
only system other than SubDomain capable of confin- 
ing sub-process components, in that Java classes are 
typically smaller than the host OS processes. Natu- 
rally, the Java 2 security model does not apply to non- 
Java native executables. 


chroot Jail 


The chroot() system call? makes the argument 
directory be the effective root directory, 1.e., “/” for 
the invoking process. The point of this operation is 
that the file system domain for the affected process is 
now limited to the contents of the argument directory. 
Any files that the application needs to access must be 
placed inside the chroot directory, or the access will 
fail. 


The chroot technique is a popular form of con- 
finement, in large part because standard kernels (e.g., 
Linux) support it. However, chroot has defects in all 
three of the dimensions a security enhancement should 
address: 

Security: chroot jails are resistant to oblivious 
attempts to escape the jail, 1.e., attempts to 
access files that are not accessible within the 
jail. However, if the attacker can execute their 
own code within the chroot jail, it is fairly easy 
to break the jail and access outside files. Thus 
jailed programs generally cannot be trusted 
with strong privileges, i.e., 1t 1s insecure to 
depend on chroot to confine a root process. 

Compatibility: Each chroot’d program must have 
the necessary components of the file system 
replicated within its jail, which 1s problematic if 
the program requires access to a large, complex 
set of files, 1.e., shell scripts need all invoked 
programs replicated into the chroot jail. Thus 
setting up a chroot jail can be a lot of tedious, 
complex work. The chroot technique also 
breaks programs that need to interact with other 
parts of the system. 

Performance: Because chroot jails require duplica- 
tion of all resources needed by the jailed pro- 
gram (soft or hard links could be used as escape 
routes) they consume excessive disk space and 
file system buffer cache space. 


3«man chroot” on most UNIX Systems 
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Type Enforcement 


The type enforcement work [13, 7, 8] has 
recently been extended to provide better support for 
program confinement. Kernel hypervisors [28] pro- 
vide a facility for installing small state machines that 
intercept kernel system calls and enforce a security 
policy. Such a facility can be viewed as a tool that 
could be used to build a SubDomain-like least privi- 
lege system. Fraiser, Badger and Feldman provide a 
similar tool for building security policy enforcement 
automata [23]. SubDomain provides the following key 
advantages over this technique: 


Parsimony: SubDomain is much simpler than the 
TE and DTE implementations; the SubDomain 
kernel code is approximately 1/10 the size of 
the DTE kernel patch. Simplicity is critical in 
security systems. 


Safety: The DTE Wrapper system [23] allows code 
to be inserted into the operating system to per- 
form mediation. While this is a powerful tech- 
nique, it is also dangerous: malicious DTE 
wrapper code could just as easily be inserted. In 
contrast, SubDomain profiles are easy to 
inspect to determine the security implications 
of updating a SubDomain profile. Furthermore, 
it 1s strictly safe to install a SubDomain profile 
where none existed before, because SubDomain 
strictly limits program privileges.‘ 

Application-Specific Mechanisms 


Various application environments provide their 
own least privilege-like mechanisms. For instance, the 
PERL interpreter includes a facility known as “‘taint’’, 
in which input provided to the PERL script cannot be 
used to formulate an action (1.e., system() operation) 
unless it has been “adequately” inspected by the 
PERL script [41]. PERL also includes a “‘safe PERL” 
facility, where in the programmer can specify a set of 
PERL operators that the script may not use. 


Another application-specific least privilege 
mechanism js the notion of “‘wrappers.”’ For example, 
CGI Wrappers [31] causes a CGI script to be run with 
the user-ID of the script owner, rather than the user-ID 
of the web server. Combined with the synthetic user- 
ID notion described previously, CGI Wrappers can 
construct a least privilege environment for CGI 
scripts. 


PACLs: Program-based Access Control Lists 


We believe PACLs [42] to be the first instance of 
an access control system based on the program per- 
forming the operation. The PACL system 1s the exact 
dual of the SubDomain notion: files have an access 
control list that enumerates programs that are permit- 
ted to operate on that file. A simulated PACL system 
was built and evaluated, but an actual PACL system 
was never finished. 


4This observation due to Blaine Bumham. 
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Status & Availability 


The implementation is not complete with respect 
to the description in this paper. 

e The absolute and _ relative sub-domains 
described earlier are not complete: child pro- 
cesses either inherit the parent’s profile, or use 
their own profile if one is specified. 

The multiple modes of requiring SubDomain 
confinement described previously is only par- 
tially implemented. The implementation cur- 
rently supports “‘paranoid”’ mode where all pro- 
cesses must have SubDomain confinement, and 
“open”? mode, where only the programs that 
are specified are confined by SubDomain. 
SubDomain is implemented for Linux, and is available 
from http://immunix.org . The kernel enhancement 
portion is licensed under the GPL, and the non-kernel 
portions are proprietary to WireX but available for 
free for non-commercial use. 


Conclusions 


Vulnerable software is a major security problem, 
mandating constant system administrator attention to 
keep systems up to date with vendor-supplied security 
patches. This is especially problematic for complex 
Internet servers, which are required to provide exten- 
Sive services to anyone on the Internet. Some form of 
confinement mechanism to approximate least privilege 
is the generic solution, but often imposes more costs 
than administrators deploying in “internet time” can 
bear. Our SubDomain confinement mechanism 
advances over previous confinement work, simpli fy- 
ing both implementation and administration overheads 
by confining programs instead of users. 


This approach enables SubDomain confinement 
to be packaged with programs, in contrast with con- 
finement mechanisms that are bound to the system. 
SubDomain also provides fine-grained protection, 
confining software components finer than the host OS 
process, providing the unique capability to protect 
potentially vulnerable server modules such as 
Microsoft’s Front Page Extensions to the Apache web 
server. We have implemented and tested the system, 
showing that it provides all three essential properties 
of a security enhancement: enhanced security, soft- 
ware compatibility, and preserved performance. 
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ABSTRACT 


NOOSE (Networked Object-Oriented Security Examiner) is a distributed vulnerability 
analysis system based on object modeling. It merges the functionality of host-based and network- 
based scanners, storing the results into several object classes. The remote agents are implemented 
as dynamically extended PERL agents. NOOSE is able to collect vulnerabilities from a variety of 
sources, including outputs from other vulnerability analysis programs (e.g., Muffet’s CRACK), 
collecting information from systems that may or may not have cooperative agents on them. 
Communication is based on a secure, reliable datagram protocol implemented as a set of PERL 
object classes. Unlike some vulnerability systems, NOOSE presents the vulnerability information 
as an integrated database, showing how vulnerabilities may be combined into chains across 
multiple accounts and systems. It understands unconditional vulnerabilities (1.e., stack-overflow, 
password guessing) along with conditional (Trojan horse, rlogin, and NFS access). Conditional 
vulnerabilities gain limited or privileges if conditions exist, such as access to specific accounts. 
The information is presented as an object-oriented "spreadsheet" format, allowing the security 
manager to explore vulnerabilities at whim. Once complete, the vulnerability analysis can move 
both forwards and backwards interactively, showing both what a selected account can attack, as 
well as showing who can attack a selected account. Besides vulnerability analysis, the system can 
intelligently verify the installation of security patches, dynamically installing missing patches. 
NOOSE is therefore a flexible prototype, able to provide a subset of the functionality of COPS, 
SATAN and TRIPWIRE, yet because of the object model, be used for developing new paradigms, 
such as reacting to intrusions, information warfare, and survivability management systems. 

Problem Statement e Vulnerability chains composed of one or more 
conditional vulnerabilities across multiple sys- 
tems aren’t evaluated. 

e The results of host-based scanning is not inte- 
grated with the results of network-based scan- 
ning. 

Some systems, such as name servers or file 
servers, need stronger protection than clients. 
Likewise, some accounts, such as _ system 
administrators, are key elements in protecting 
the overall security. However, vulnerability 


This paper discusses limitations in Vulnerability 
Analysis systems such as COPS, SATAN Tiger, RSS 
and ISS. For convenience, these systems will be 
referred to as VA systems. In this paper, a vulnerability 
is a potential path to break into someone’s account to ‘ 
elevate their privilege. This paper also discusses vul- 
nerability chains, which is defined to be two or more 
vulnerabilities, that can be executed in sequence. An 
example of a chain is using NFS to insert a Trojan 
horse into a directory, which can be executed by a sys- analysis systems ignore these different threat 
tem administrator, to gain root access to a file server. potentials. 

Once this key account has been breached, the group of ° Many security systems characterize potential 


related clients become vulnerable because of the rela- 
tionship between servers and clients. This collection 
of systems will be called a workgroup. In a large facil- 
ity, there may be dozens or hundreds of workgroups, 
with separate administrators. 


The goal of this paper is to find a way to identify 
and eliminate these vulnerability chains among and 
between workgroups. The author believes current VA 
systems have difficulty in doing this because of the 
following reasons: 

e Some potential vulnerabilities are accepted as a 
matter of policy, and the convenience value 
overrides the potential risk. These are consid- 
ered conditional vulnerabilities, because other 
conditions determine how risky it 1s. 
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failures with a simplistic red/yellow/green indi- 
cator or a numbered scale with 5 values. This 1s 
too coarse a measurement to be useful. The 
simple statement “NFS is insecure” is often 
ignored, while a report that identifies the pre- 
cise directory that can be used to compromise 
the root account is more likely to be fixed. 


The author feels that many current VA systems 
don’t properly analyze the consequences of an intruder 
who breaches a firewall, or an insider attempting to 
gain access. Systems have complex relationships 
between clients and servers, and the compromise of a 
single server can allow hundreds of clients to be com- 
promised. Servers can also be clients of other servers, 
and clients can be used to compromise other servers. 
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Likewise, accounts have complex inter-relationships, 
and system administrator accounts require special pro- 
tection. 

The vulnerability chain that was mentioned ear- 
lier, for instance, makes it trivial to break into an 
account without detection. Using low-level NFS calls 
(e.g., Leendert van Doorn NFSSHELL) typically 
requires no special privileges and allows access to any 
file on an NFS-exported file system not owned by 
root, such as those owned by a system administrator. 
If that account is not directly accessible, a Trojan 
horse can often be used. Since these attack mecha- 
nisms use ordinary file access mechanisms, they are 
rarely detected. Therefore it is essential that the poten- 
tial danger be reported and repaired. Identifying this 
type of danger 1s one of the primary goals of NOOSE. 


There have been attempts to integrate informa- 
tion, but these have been limited to a common output 
format [12], or a common user interface [11]. Kuang 
[1] identified vulnerability chains, but was limited to a 
single system. NetKuang [5] is one attempt to correct 
this. NOOSE uses a second approach. 


Background 


This implementation was based on our experi- 
ence with an earlier Expert Fault Manager system 
[13], where the importance of relationships was 
emphasized. The author decided to apply what was 
learned while measuring security risks. However, sev- 
eral custom agents were on each system, and they 
needed to be upgraded manually. The communication 
system suffered from deadlocks occasionally. The core 
components were enhanced and applied to a vulnera- 
bility analysis prototype [14]. The work was influ- 
enced by the author’s program for analyzing Trojan 
horses on UNIX systems, and various tools from Pur- 
due [2, 3, 4]. This project was started in January of 
1996, with funding was provided by Lockheed Martin 
and L-3 Communications. This paper describes the 
lessons learned from this prototype. 


The author choose the name NOOSE because of 
the concept of drawing a circle around a set of arbi- 
trary systems, and identifying vulnerabilities within 
this set, using OO techniques. 


Description of System 


The NOOSE system is written using PERL, cho- 
sen because of the power of the language, string pars- 
ing, and the ease of prototyping, as well as being able 
to potentially migrate code from COPS and 
SATAN/SAINT variants. NOOSE is implemented 
with 26 PERL-based object classes. A PERL agent 
exists on all systems cooperating with NOOSE. These 
agents talk to NOOSE using a centralized dispatcher, 
which in turn talks to an Information Warfare (IW) 
module. A Graphic User Interface (GUI) is written in 
PERL and TK, and provides a spreadsheet-like inter- 
face to the information contained in the IW module. 
The overall architecture can be seen in Figure 1. 


370 


Barnett 


Auxiliary files exist that contain information 
about patches, operating systems, and PERL modules 
to be uploaded to the agent. 





Figure 1: NOOSE architecture. 


The agents just contains enough information to 
listen for new commands. New subroutines can be 
uploaded, and the revision tracked. If modules require 
other modules that are undefined, this is reported as an 
error. Therefore the agents are small, easy to install, 
and never need to be updated manually, which reduces 
maintenance costs. A simple dependency system was 
put in place, to identify required modules. However, 
the implementation used a single file for each operat- 
ing system variant, and uploaded the entire file to the 
agent on demand. Therefore the dependency feature 
was rarely used. 


The agent will run on a Windows NT box, but no 
vulnerabilities are currently gathered. 


Object Classes 


The primary goal of object-oriented program- 
ming is the ability to reuse object classes, lowering 
cost of development. Therefore our goals was to 
develop an object model that can support multiple 
algorithms. A data structure that can be used with two 
diverse algorithms has a better chance of being used 
by future algorithms. Object modeling provides a con- 
cise mechanism to document the data structures used 
in a system, often on a single page. 


Object Model 


The implementation contains of the following 
object classes: 
¢ Communication (9 classes) 
e System-related (1 Primary base class, 1 sec- 
ondary base class, 13 sub-classes) 
e GUI (2 classes) 
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System-related object classes 


The NOOSE system uses the following object 
model, as shown in Figure 2. The 13 different object 
classes maintain information about remote systems, 
corresponding to their state: 

e Patch 
e Host 
e Signature 
e Operating System 
e Account 
¢ UID 
e Vulnerability 
e UNIX Resource (which has four sub classes) 
e File 
e Link 
e Directory 
e Missing 
FileState 
e PatchState 


All system-related objects have a unique name. 
NOOSE uses a simple ASCII string composed of the 
following parts: 


NOOSE 


Object Class 
Host responsible for object 
Object-specific information 


Examples are 
account/pluto/smith 
uid/pluto/214 


host/pluto 
dir/pluto//etc/mail/sendmail.cf 


This provides a simple, extensible way to create 
new object types, as well as a simple method of locat- 
ing the authority of the object (in this case, the host 
that must be queried to get information about that 
object.) In our communication paradigm, instances of 
objects resided on different hosts. 


Relationship Superclass 


All of the 13 system-related classes are derived 
from a base class that provides object lookup, cre- 
ation, deletion, as well as relationship creation, query- 
ing, and navigation. This relationship or association is 
critical to the implementation. It creates one-to-one, 
one-to-many, and many-to-many relationships between 


NOOSE Object Model (OMT Notation) 
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Figure 2: NOOSE object model. 
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any two system-related classes. Creating a one-to- 
many relationship between two instances (e.g., there 
are several files related to a patch) merely requires 
associating two object references with the method call: 


SpatchObject->one_to_many ($fileObject) ; 


No additional declarations are necessary. This simpli- 
fied modi fications of the object model. Other methods 
are one_to_one, many_to_one and many_to_many. If 
a one-to-one relationship is created,and a second rela- 
tionship to the same class is added, an error is gener- 
ated, suggesting a one-to-many relationship be used 
instead. The base class also provides ways to obtain, 
find, test, search and integrate related objects. It can 
be used as a collector object, and simplifies algorithm 
development significantly. The code fragment below 
demonstrates the methods as it finds and tests all of 
the required patched files corresponding to a revision 
of an operating system; see Listing 1. 


These system-related classes, besides used to 
store information about the objects, allow algorithms 
to be easily constructed based on the relationship (or 
association) between objects. Often the relationship 
between two classes need not have a specific name, 
and the relationship is obvious from the context. 
Some objects have very few attributes, as the primary 
purpose is collection and navigation. 


Two key classes are Accounts and Vulnerabili- 
ties. An account object is a username/hostname pair. 
The vulnerability object always has relationships to 
two accounts (except when the account has been 
removed, in which case it refers to the UID object 
class, which corresponds to the user ID number. In our 
system, a vulnerability is a potential mechanism to 
allow someone to go from one account to a different 
account. 


The first relationship is to the account that can be 
compromised (the attackee), while the second shows 
the account that can compromise the first account (the 


#Specify host to check 
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attacher). Vulnerability objects may have a optional 
relationship to a file or directory, indicating the cause 
of the vulnerability. 


Wildcards in Account names 


Accounts consisted of two pieces of information: 
the username, and the hostname. NOOSE uses a “*” 
to indicate a wildcard, which may be in either the host, 
username or both. Accounts with wildcards are used to 
describe specific vulnerability classes. If an account 
had no password, then anyone on any system could 
access that account. This fictitious user is indicated by 
the account “account/*/*.” If Joe Smith’s account has 
a “+” in the “.rhosts” file, then the vulnerability can 
be initiated from the account “account/*/smith”. If 
Smith’s account on host “pluto” has a “.” first in the 
searchpath, then this account can be compromised by 
the “account/pluto/*” account, which means anyone 
on host pluto. This simple naming convention can be 
used to describe the starting attack point of any vul- 
nerability. In the case of vulnerabilities within a group 
or netgroup privileges, multiple vulnerabilities are cre- 
ated, with the attacking accounts expanded to the com- 
plete list of individuals within this group. 


Vulnerability Chains 


A vulnerability chain occurs when multiple vul- 
nerabilities can be used to achieve a particular goal (or 
in this case, an account). Figure 3 shows such a chain. 


In this case, assume a hacker can break into the 
[pd account on host hosta because it was missing a 
security patch. Next the attacker uses NFSSHELL to 
access the home directory of account hostb/smith. This 
account has write privileges in /usr/local/bin and a 
Trojan horse is created. The backup account has this 
directory in the searchpath, and executing the Trojan 
horse compromises the account. Once done, the user 
may gain access to the root account on a file server, 
which allows access to all of the clients. 


Sos = Os->fetch(Shost->get_os_type); # Find the OS type and revision 
foreach $patch ($os->get_many("Patch") { # get the patches for the OS 
fCat the files: anc luded in each patch cluster 


foreach stile. (Spatceh-7get. many( File") 


{ 


4## Files have more than one Signature - depends on OS rev 
foreach Ssignature (S$file->get_many("Signature")) { 
# Only look at those that match the OS and revision 


if (Ssienature-7has (Sos) ) 4 
Ssignature->verify ( 
host=>Shost, 
fa be—>S fi he, 
patch Sparch): 
| 
} # Signatures 
} # files 
} # Patches 


found a2 maton 


Listing 1: Testing patched files. 
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Communication classes 


All network communication is comprised of 
objects and methods, which are specified as ASCII 
strings. The dispatcher looks at the object, determines 
the host to send the message to, and sends the infor- 
mation to the clients using a protocol layered on top of 


¢ 
HII 
HOSTA SERVER 


lo 
ag MER = 
= 


| |HosTabacu " HiT 


HOSTA 














Clients 


Figure 3: Vulnerability chain. 


User Datagram Protocols. The original Expert Fault 
Manager implementation limited communication to 
small packets, and sent information as simple ASCII. 
Packet loss and security wasn’t addressed. However, 
as larger pieces of information was needed, and the 
system suffered from deadlocks, a redesign was 
needed. A decision was made to provide the old 
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function as a base class, and creating sub-classes that 
add features. 


The original system used UDP because of the 
need for realtime predictable responses, and the desire 
to extend the protocol to use multicast and broadcast 
transports in the future. Three low-level UDP classes 
are broken into UDP Client, UDP Server, and UDP 
Common functions. All three are sub-classed to pro- 
vide reliability and packet assembly and disassembly. 
These are again sub-classed to provide secure commu- 
nication, using symmetric keys for encryption. The 
security used is weak, as keys are distributed manu- 
ally, and reused for each session. 


The communication object class provide a sim- 
ple send and receive function. A system acting as a 
relay or filter, which is both a client and server, while 
avoiding deadlocks, merely needs the PERL fragment 
in Listing 2. 

The wait method always waits on the socket 
associated with the object to the left of the arrow. 
Additional objects, corresponding to sockets, may be 
included. This single event loop, combined with time- 
outs and the test for pending data, was important in 
preventing deadlocks. 


The communication classes contains options for 
verifying the reliability of the communication. Errors 
(i.e., dropped packets) can be purposely created by 
dropping a percentage of the packets pseudo-ran- 
domly. This was used to verify the reliability of the 
communication subsystem. Other options specify 
buffer sizes, timeouts, status, and counts of packets 
sent and received. The higher level routines reassem- 
ble packet fragments, and retransmit missing packets. 
The methods used for the reliability and secure sub- 
classes are the same as the base class as far as the 
application is concerned. However, there were several 
dozen private used in the implementation. 


Several protocols were attempted to provide reli- 
able communication. ‘To validate the protocol, a 


my $client = udpsecclient->new( # act as a client to another server 


pOrt—- sport, 
machine=>Smachine, 
security™Slevel) ; 


my $server = udpsecserver->new( # set up our own server 


port Sport: 
security Slevel); 
while (1) { 


Sserver->wait($client); # Wait for activity on either socket 
if (Sserver->pending) { # Our server gets data 


Sclient->send (S$server->receive) ; 


} 


4 send to other server 


if ($client->pending) { # Our client gets results from server 


Sserver->send ($client->receive) ; 


} 


## send back to our client 


Listing 2: Simple client/server model. 
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Design of Experiment (DoE) methodology was used, 
varying buffer sizes, size of files, and number of 
simultaneous connections. Developing the reliable 
protocol was difficult. While the DoE did not clearly 
indicate the optinum parameters, a reliable protocol 
was selected that used a common (and symmetric) 
send/receive method for clients and servers. The 
sender breaks up the message into smaller pieces, and 
sends them out sequentially. The receiver responds 
with a positive acknowledgment, indicating all packets 
were received, or a negative acknowledgment, which 
requests the sender to re-transmit the missing packets. 
Once a positive acknowledgment from the receiver 
has been received, the sender transmits an acknowl- 
edgment of the acknowledgment and changes states. 
This same method is used by both sides to communi- 
cate. Therefore the client requests informations from a 
server, a minimum of 6 packets are transmitted, and 
more if packets are lost. The primary difference 
between the client and the server classes was the 
states: a server is finished after sending information, 
and the client is finished after receiving information. 
Caching was easily added, with answers based on the 
combination of the object and the method. The results 
are saved if there were no errors. 


Functionality 


The system can potentially accept vulnerabilities 
from any source, but a parser must be build to extract 
the vulnerability type and account information. The 
system, as currently implemented, performs the fol- 
lowing vulnerability checks: 


Trojan Horse 

NFS 

RLOGIN/SSH 

Output of Alec Muffet’s CRACK program 
Missing security-related patches 


The first three operate on account objects (once 
for each account), while the last two operate on a host 
basis (once for each host). 


The Trojan horse program, uploaded to the 
agent, finds the shell of the user and examines the 
appropriate start-up files. It parses the files, and keeps 
track of files that are sources, as well as the value of 
variables. When searchpaths are specified or modified, 
the values of these variables are used to determine the 
potential searchpath. If branches are taken, the pro- 
gram assumes both paths are used, so the searchpath 
examined by the program is the superset of the actual 
searchpath. The program, for safety, doesn’t evaluate 
commands within back-quotes. Instead, a predeter- 
mined set of commands are evaluated once on each 
host, as specified by the program, and if one of the 
users has this string in their searchpath, the pre-deter- 
mined value is inserted into the string. If it is not 
known, it is ignored. Consider the following C shell 
fragment: 
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if ue =f) local: say 
Loealipathir=C /Lhocal/- aren) panies 
else 
hocalpath = j/isr/ “arch / bin. 
/usr/ locally “arch 7 bin?) 
endif 
set path = ( Slocalpath Spath ) 


All three directories will be examined in addition 
to the default value of the searchpath, assuming the 
directories exist. The software also detects recursive 
loops, and handles them to a depth of two, and aborts 
if a loop is detected. Using this, it is possible to get a 
searchpath for each account. This, in turn, 1s used to 
measure the potential for Trojan Horse attacks on all 
accounts. Currently the system only examines the per- 
missions of the directories, as well as the permission 
of the parent directories, and symbolic links. The per- 
mission of the files inside the directory are not exam- 
ined, unlike the author’s Trojan checking program. 


The algorithm asks the agent to extract all of the 
user and group information from the host, 1f necessary. 
Then it creates a set of account objects matching each 
found. Originally, each group and account was asked 
individually, but a new dispatch method was created to 
retrieve all of the account information in a single 
query for efficiency. 


When one account is selected for the vulnerabil- 
ity check, the IW module asks the agent to get the 
searchpath, which returns a list of directories by name. 
The IW module creates instances of these objects as 
needed in its own database, and asks the agent to list 
who has write permission for each of the directories (if 
this has not been asked before). Because each object 
has a name that is unique, and also specifies the host 
responsible for the information, instances of objects 
are externalized, and the object name and method is 
passed to the agent in plain text, and the results is also 
in plain text, fragmented and encrypted according to 
the communication object class. By examining each 
directory for user-, group- and world-write permis- 
sion, the agent can return a list of accounts that have 
the ability to write to the directory specified. A special 
type of object is needed to correspond to missing 
directories. This checks for the potential of someone 
being able to create a directory in the future. 


When a Trojan Horse is found, one or more vul- 
nerabilities are created that specifies the victim and 
the potential attacker(s), as well as the file or directory 
that caused the problem. In the case of a vulnerability 
by group-write permission, multiple vulnerabilities are 
created, listing everyone in the group as a unique 
attacker. If a directory is world-writable, or the user 
allows the current directory to be in the searchpath, 
the attacker is the wildcard account on this machine. 


UID’s are used when a directory is owned by an 
account that no longer exists (1.e., has no name associ- 
ated to it.) 
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NFS 


When an account is examined for NFS vulnera- 
bility, the algorithm determines if the file is on a NFS 
client or a NFS server. Examined on a client, the 
“attacker” is the root account on the server where the 
directory is exported, as well as the root account on 
the client. However, on a server, the system examines 
the export list, and the netgroup information, and 
specifies all of the accounts that have the ability to 
modify the files. In other words, when examining 
“smith” on “pluto”, if “smith” on “neptune” has 
write permission, then “root” on “neptune” does as 
well. The system also examines the NFS server if the 
client port numbers must be less than 1024, indicating 
a privileged account. If not, then the attacking account 
is the wildcard account “anyone” on “neptune”. If the 
directory is exported to the world, the attacker is iden- 
tified as “anyone” on “anyhost,” indicating that any- 
One on any machine that can access the system can 
break into the account. 


RLOGIN 


The agent examines the system configuration for 
/etc/hosts.equiv and $HOME/rhosts as well as the 
SSH equivalent files to determine which counts have 
access to the selected account. NIS netgroups and “+” 
in the .rhosts file are understood, and appropriate vul- 
nerabilities are created. 


CRACK 


The system parses a file generated by CRACK, 
and looks for a matching hostname. When found, it 
creates a vulnerability between the “anybody” on this 
host to the account whose password was guessed. 


Checking System Patches 


The IW module first reads a series of files that 
contain a one-line summary per file associated with a 
patch. The OS type, revision, architecture, file path, 
patch ID, size, and MDS value of each file is speci- 
fied. This file is created automatically for Solaris sys- 
tems using a shell script that weekly retrieving the cur- 
rent patch status. The IW module, while reading this 
file, creates Patch, File, OS and Signature objects. 
These must be separate objects because a file may 
have multiple signatures depending on the OS, revi- 
sion, and patch. Also created is a FileState and Patch- 
State object which corresponds to the actual state of a 
file and patch on a particular host, instead of the gen- 
eralized information, valid for all systems of the same 
revision. In other words, PatchState and FileState have 
an association with a specific host. 


The system gets the list of files for each patch 
and examines the signature of each file if it has per- 
mission. Five results are possible: (1) correct revision, 
(2) wrong revision, (3) file does not exist, and (4) 
insufficient privileges to read file and (5) system can- 
not execute the external MD5 program to check 
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signature. The system can form a conclusion based on 
partial information. If it cannot read one file of a 
patch, and a second is the wrong revision, it concludes 
the patch has not been applied. If it is uncertain 
because it cannot read all of the files, but the rest are 
correct, it indicates the patch might be applied. There- 
fore determining if patches have been applied is accu- 
rate even if someone replaced a file after a patch has 
been applied, or modified any of the utilities (i.e., 
showrev). Missing patches create different vulnerabil- 
ities, depending on permissions of the un-patched file. 
This is simplistic, and uses the set-uid permissions and 
owner information to identify the attacked account. If 
the file is a library, the attacked account is considered 
to be the root account. A future version should use an 
internal MDS5 checksum utility to prevent tampering 
with the executable. 


Using the GUI, it 1s possible for the security offi- 
cer to see the state of the patch and associated files. If 
desired, new patches can be uploaded to the system, 
and applied. 


User Interface 


The GUI (Figure 4) presents the information in 
three sections, General operations, host-specific opera- 
tions, and account-specific operations. The general 
commands are a series of buttons that print out sum- 
maries, load the patch database, dump the database, 
and interface to other applications for advanced analy- 
SiS. 


The host specific operations include the follow- 
ing actions: 

¢ Uploading the PERL modules. 

e Querying the revision of the current modules 

e Fetching the user and group information 

e Listing all of the UID’s on a host 

e Listing all of the accounts on a host. 

e Scanning the output of CRACK, merging new 
vulnerabilities 

e Scanning the system for missing patches 

e Displaying the results of the patch analysis 


Displaying the UID or accounts presents the 
information in a scrolling list below, which can be 
used for the account-specific operations. Typically 
accounts are displayed in the scrolling list, and one or 
more accounts can be selected, and the account-spe- 
cific methods can be used. Operations iterate over 
each item selected. If no account is selected, the 
account in the Argument: location is used. Typically 
the account that is of primary interest is copied and 
pasted into this for convenience. 


The account-specific methods include: 

e Trojan — Perform a Trojan Horse scan on the 
selected account 

e NFS — Perform a NFS vulnerability check 

e RLOGIN ~ Perform checks on 
rlogin/rsh/rcp/ssh 

e Attacker — shows everyone who can attack the 
selected account 
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e Attackee -— shows everyone the selected 
account can attack 

e Show Vulnerabilities — shows account-specific 
vulnerabilities 


Dispatcher and objects 


The dispatch system was reused from an earlier 
project, using a version of PERL that did not support 
OO. The earlier project used “pseudo-object-oriented” 
techniques. That is, the dispatcher used strings to iden- 
tify objects, methods and parameters. Based on its 
table, it would select a protocol, host and port an com- 
municate with the remote system using the communi- 
cation object classes. This included methods for 
query/response, uploading files and patches, and ask- 
ing the remote agent to evaluate and execute PERL 
code. There was a weak correlation between these 
pseudo-objects and the object class used by the IW 
module. 


Problems Encountered 


There were four significant problems with the 
implementation: 
e This prototype evolved from an earlier system, 
and the object classes were weakly integrated 
with the remote execution of commands. 
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never asked. Answers could be cached on the 
agent, or in the dispatch, or integrated into the 
database. Timeouts could occur anywhere. A 
redesign with a unified view of the remote 
information is needed. 

Secondly, retrieving the information was often 
piecemeal. Sometimes objects were created 
merely as collection objects and navigation 
objects. Accounts were created when vulnera- 
bilities were found, and attributes and associa- 
tions about these objects would often be left 
undefined. Therefore a large part of the code is 
testing for the existence of information, and 
performing queries to fill in missing informa- 
tion if needed. 

Any distributed system is unreliable and asyn- 
chronous. Therefore any remote query could 
fail, and failures had to be handled. A better 
design would have asynchronous gathering of 
information, and _ well-defined algorithms 
driven by an expert system, reporting on prob- 
lems. 

The design of the GUI allowed gathering and 
traversal of the information in any order 
desired. There was no pre-determined order in 
the information gathering process. This pro- 
vided flexibility, but added to the complexity. 
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Agents and their methods could have multiple 
states. Methods could be undefined or out of 
date. Queries could be pending, obsolete, or 
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These problems caused the code to be bulky and 
inelegant. Everything worked, but similar code had to 
be inserted in multiple places. 
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Figure 4: NOOSE GUI. 
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The second biggest difficulty was the develop- 
ment of the secure, reliable datagram-based protocol. 
Developing distinct classes between the server and 
client was essential, but finding the common methods 
that supported three types of communication (raw, 
reliable, secure) was difficult. A minor change in the 
algorithm could easily cause a deadlock. 


Conclusions 


As in our earlier project [13], an important ele- 
ment to these algorithms is the ability to traverse data 
structures using associations between objects. This 
allows algorithms to use the context surrounding 
objects while making decisions about single objects. 
This also stimulates new algorithm development, 
while reusing existing code. In some cases, the objects 
needed few attributes, as 1t was used primarily as a 
navigation point in the data structure. 


The system has reasonable performance. Large 
servers with 2000 accounts could have each account 
examined in about 30 minutes. A large proportion of 
this time is believed to be associated with auto mount- 
ing the directories found in users search paths. 


Because much of the software required handling 
missing data, and undefined values, a multi-threaded, 
asynchronous programming model is needed, com- 
bined with an object model that unifies the data str uc- 
tures of the agent and system. 


The three-way handshake for reliable datagram 
communication is important for deadlock protection 
and obvious in hindsight. The symmetric nature 
allowed code reuse, and therefore simplified the object 
classes. Using this, it was possible to transmit patch 
clusters of 70 megabytes. Occasionally timeouts 
occurred where both sides were waiting, but the sys- 
tem typically recovered. Creating communication 
object classes designed for sub-classing allows future 
extensions while retaining the same API. 


Future Directions 


The communication classes can be extended is 
several ways. A better key distribution system can be 
inserted, as well as alternate encryption algorithms. 
Also, a subclass can be created that combines multiple 
lightweight messages in a_ single  authenti- 
cated/encrypted packet. A mechanism for proxy 
agents can be added, as well as a means to locate 
agents by broadcast. Threaded asynchronous commu- 
nication would simplify code development, as would 
uniform methods for testing and retrieving remote 
attributes. 


Objects are currently stored in memory only, or 
dumped to an external file for analysis. An ASCII rep- 
resentation of the database is about 1 Megabyte in size 
for 2000 vulnerabilities. A persistent database is 
desirable, especially to a relational database. 


Alternate mechanisms of viewing and analyzing 
vulnerabilities is desired. Several simple algorithms 
for categorizing vulnerabilities are suggested: 
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e Identify files with the largest number of associ- 
ated vulnerabilities. 

e Counting the number of paths between any two 
accounts. 

e Identify all of the accounts that can potentially 
break into a selected account. 

e Identify the worst case result of a single com- 
promised account. 


The author feels the concepts can be used for 
advanced policy management systems, as security can 
be measured more accurately, and relationships can be 
constructed between files and services. 


Summary 


The author strongly feels that object modeling is 
essential to writing next-generation security algo- 
rithms, allowing a single database to be used for 
diverse algorithms. This single database merges 
together the base functionality of host-based scanners, 
network-based scanners, file tampering and intelligent 
patch management systems. The author hopes the 
object model will be useful to others developing sim1- 
lar applications. 


The author believes that this implementation 
shows several unique traits. The implementation sim- 
plifies prototyping new algorithms, and allows reusing 
data for multiple applications. Adding the patch man- 
agement software was simpler than the core communi- 
cation classes. The agent structure simplifies support. 
The object class for communication can be extended 
in many ways. The use of the vulnerability and 
account object class provides a simple and elegant, yet 
powerful way to integrate information from multiple 
sources, as we feel there is great Notential and flexi- 
bility to this mechanism. 
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THE USENIX ASSOCIATION 


Since 1975, the USENIX Association has brought together the community of developers, programmers, system 
administrators, and architects working on the cutting edge of the computing world. USENIX conferences have 
become the essential meeting grounds for the presentation and discussion of the most advanced information on 


new developments in all aspects of advanced computing systems. USENIX and its members are dedicated to: 
e problem-solving with a practical bias 

e fostering innovation and research that works 

¢ communicating rapidly the results of both research and innovation 
e providing a neutral forum for the exercise of critical thought and the airing of technical issues 


SAGE, the System Administrators Guild 


The System Administrators Guild, a Special Technical Group within the USENIX Association, is dedicated to the 
recognition and advancement of system administration as a profession. To join SAGE, you must be a member of 


USENIX. 


Member Benefits: 


e Free subscription to ;login:, the Association’s magazine, published eight times a year, featuring technical 
articles, system administration tips and techniques, practical columns on Perl, Java, Tcl/Tk, and Open Source, 
book and software reviews, summaries of sessions at USENIX conferences, and Standards Reports from the 
USENIX representative and others on various ANSI, IEEE, and ISO standards efforts. 


Access to ;login: on the USENIX Web site. 
Access to papers from the USENIX Conferences and Symposia, starting with 1993, on the USENIX Web site. 
Discounts on registration fees for the annual, multi-topic technical conference, the System Administration 


Conference (LISA), and the various single-topic symposia addressing topics such as security, Linux, Internet 
technologies and systems, operating systems, and Windows—as many as twelve technical meetings every year. 


http://www. usenix.org/membership/specialdisc.html for details. 


Supporting Members of the USENIX Association: 


Addison-Wesley 
Earthlink Network 
Edgix 

Interhack Corporation 
Interliant 

Lucent Technologies 


Microsoft Research 

Motorola Australia Software Centre 
Nimrod AS 

O’Reilly & Associates Inc. 

Sams Publishing 

Sendmail, Inc. 


Supporting Members of SAGE: 


Collective Technologies 
Deer Run Associates 
Electric Lightwave, Inc. 
ESM Services, Inc. 
GNAC, Inc. 

Mentor Graphics Corp. 


Microsoft Research 

Motorola Australia Software Centre 
New Riders Press 

O’Reilly & Associates Inc. 

Remedy Corporation 


For more information about membership, conferences, or publications, 


see http://www.usenix.org/ 


or contact: 


Discounts on the purchase of proceedings and CD-ROMs from USENIX conferences and symposia and other 
technical publications. 

The right to vote on matters affecting the Association, its bylaws, and election of its directors and officers. 
Savings on a variety of products, books, software, and periodicals: see 


Smart Storage, Inc. 

Sun Microsystems, Inc. 

Sybase, Inc. 

Syntax, Inc. 

Taos: The Sys Admin Company 
UUNET Technologies, Inc. 


RIPE NCC 

Sams Publishing 

SysAdmin Magazine 

Taos: The Sys Admin Company 
Unix Guru Universe 


USENIX Association, 2560 Ninth Street, Suite 215, Berkeley, CA 94710 USA. 
Phone: 510-528-8649. Fax: 510-548-5738. Email: office @ usenix.org. 
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